load_file_to_objectstore

Write a file to a container on the objectstore.

Use ENV:
export OBJECTSTORE_PASSWORD=********** to add the password to your environment before running this command script.
Example commandline code:
load_file_to_objectstore config.ini objectstore data/test.csv Dataservices/aanvalsplan_schoon/crow

usage: load_file_to_objectstore [-h]
                                config_path config_name filename
                                container_path

Positional Arguments

config_path Define the full path of the config file, for example /path_to_config/config.ini or config.ini
config_name Specify the name in the config.ini file, for example objectstore
filename Specify the name of the file
container_path Specify the full container and prefix path where to upload to, Dataservices/aanvalsplan_schoon/crow

functions

datapunt_processing.load.load_file_to_objectstore.check_existence_object(connection, container_path, filename)

Check if the file is present on the objectstore container_path,

Args:
  1. connection = Objectstore connection based on from helpers.connection import objectstore_connection
  2. container_path = Name of container/prefix/subfolder
  3. filename = Name of file, for example test.csv
Returns:
  • ‘The object was successfully created’
  • ‘The object was not found’
  • ‘Error finding the object’
datapunt_processing.load.load_file_to_objectstore.get_object(connection, container_path, filename, output_folder)

Download file from objectstore container.

Args:
  1. connection: Objectstore connection based on from helpers.connection import objectstore_connection
  2. container_path: Name of container/prefix/subfolder
  3. filename: Name of file, for example test.csv
  4. output_folder: Define the path to write the file to for example app/data when using docker.
Returns:
A file from the objectstore into the specified output_folder.
datapunt_processing.load.load_file_to_objectstore.parser()

Parser function to run arguments from commandline and to add description to sphinx docs.

datapunt_processing.load.load_file_to_objectstore.put_object(connection, container: str, filename: str, contents, content_type: str) → None

Upload a file to objectstore.

Args:
  1. container: path/in/store
  2. filename: your_file_name.txt
  3. contents: contents of file with use of with open(‘your_file_name.txt’, ‘rb’) as contents:
  4. content_type:’text/csv’,’application/json’, … Is retrievd by using the mime package.
Returns:
A saved file in the container of the objectstore.
datapunt_processing.load.load_file_to_objectstore.upload_file(connection, container_path, filename_path)

Upload file to the objectstore.

Args:
  1. connection = Objectstore connection based on from helpers.connection import objectstore_connection
  2. container_path = Name of container/prefix/subfolder, for example Dataservices/aanvalsplan_schoon/crow
  3. filename_path = full path including the name of file, for example: data/test.csv

Uses mime for content_type: https://stackoverflow.com/questions/43580/how-to-find-the-mime-type-of-a-file-in-python

Result:
Uploads a file to the objectstore and checks if it exists on in the defined container_path.