download_from_objectstore¶
Download files from the objectstore:
download_from_objectstore config.ini objectstore aanvalsplan_schoon/crow,aanvalsplan_schoon/mora data
Use
export OBJECTSTORE_PASSWORD=**********
to add the password to your environment before running this command script.
usage: download_from_objectstore [-h]
config_path config_name prefixes
output_folder
Positional Arguments¶
config_path | Define the full path of the config file, for example /path_to_config/config.ini or config.ini |
config_name | Specify the name in the config.ini file, for example objectstore |
prefixes | Specify the names of the folders to download, separated by a comma, for example aanvalsplan_schoon/crow, aanvalsplan_schoon/mora |
output_folder | Specify the output_folder location, for example data |
functions¶
-
datapunt_processing.extract.download_from_objectstore.
download_container
(connection, container, prefix, output_folder)¶ Download file from objectstore.
- Args:
- connection: connection session using the objectstore_connection function from the helpers.connections
- prefix: tag or folder name of file, for example subfolder/subsubfolder
- output_folder = ‘/{folder}/ ‘
- Returns:
- Written file /{folder}/{prefix}/{file}
-
datapunt_processing.extract.download_from_objectstore.
download_containers
(config_path, config_name, prefixes, output_folder)¶ Download multiple files from the objectstore.
- Args:
- connection: connection session using the objectstore_connection function from the helpers.connections
- prefixes: multiple folders where the files are located, for example aanvalsplan_schoon/crow,aanvalsplan_schoon/mora
- output_folder: local folder to write files into, for example app/data for a docker setup
- Result:
- Loops through download_container function for each prefix (=folder)
-
datapunt_processing.extract.download_from_objectstore.
get_full_container_list
(connection, container, **kwargs)¶ Get all files stored in container (incl. sub-containers)
- Args:
- connection: connection session using the objectstore_connection function from the helpers.connections
- container: “name of the root container/folder in objectstore”
- Returns:
- Generator object with all containers.
-
datapunt_processing.extract.download_from_objectstore.
parser
()¶ Parser function to run arguments from commandline and to add description to sphinx docs.