download_from_objectstore

Download files from the objectstore: download_from_objectstore config.ini objectstore aanvalsplan_schoon/crow,aanvalsplan_schoon/mora data

Use export OBJECTSTORE_PASSWORD=********** to add the password to your environment before running this command script.

usage: download_from_objectstore [-h]
                                 config_path config_name prefixes
                                 output_folder

Positional Arguments

config_path Define the full path of the config file, for example /path_to_config/config.ini or config.ini
config_name Specify the name in the config.ini file, for example objectstore
prefixes Specify the names of the folders to download, separated by a comma, for example aanvalsplan_schoon/crow, aanvalsplan_schoon/mora
output_folder Specify the output_folder location, for example data

functions

datapunt_processing.extract.download_from_objectstore.download_container(connection, container, prefix, output_folder)

Download file from objectstore.

Args:
  1. connection: connection session using the objectstore_connection function from the helpers.connections
  2. prefix: tag or folder name of file, for example subfolder/subsubfolder
  3. output_folder = ‘/{folder}/ ‘
Returns:
Written file /{folder}/{prefix}/{file}
datapunt_processing.extract.download_from_objectstore.download_containers(config_path, config_name, prefixes, output_folder)

Download multiple files from the objectstore.

Args:
  1. connection: connection session using the objectstore_connection function from the helpers.connections
  2. prefixes: multiple folders where the files are located, for example aanvalsplan_schoon/crow,aanvalsplan_schoon/mora
  3. output_folder: local folder to write files into, for example app/data for a docker setup
Result:
Loops through download_container function for each prefix (=folder)
datapunt_processing.extract.download_from_objectstore.get_full_container_list(connection, container, **kwargs)

Get all files stored in container (incl. sub-containers)

Args:
  1. connection: connection session using the objectstore_connection function from the helpers.connections
  2. container: “name of the root container/folder in objectstore”
Returns:
Generator object with all containers.
datapunt_processing.extract.download_from_objectstore.parser()

Parser function to run arguments from commandline and to add description to sphinx docs.