download_from_ckan

Get data and metadata from data.amsterdam.nl, unzip if needed and put the file in a local directory. To test run this command line: download_from_catalog https://data.amsterdam.nl/#?dte=catalogus%2Fapi%2F3%2Faction%2Fpackage_show%3Fid%3D5d84c216-b826-4406-8297-292678dee13c data

usage: download_from_data_amsterdam_ckan [-h] url output_folder

Positional Arguments

url
Insert full url from main result page of dataset, for example: “https://data.amsterdam.nl/#?dte=catalogus%2Fapi%2F3%2Faction%2Fpackage_show%3Fid%3D5d84c216-b826-4406-8297-292678dee13c”

Default: “https://data.amsterdam.nl/#?dte=catalogus%2Fapi%2F3%2Faction%2Fpackage_show%3Fid%3D5d84c216-b826-4406-8297-292678dee13c”

output_folder
Specify the desired output folder path, for example: app/data

functions

datapunt_processing.extract.download_from_ckan.download_all_files(metadata, download_directory)

Download all files from metadata resources list.

Args:
  1. metadata: json dictonary from ckan with all the metadata including the resources list of all files.
  2. download_directory: path where to store the files from the files, for example data.
Result:
Unzipped and created dir filled with all data in the download_directory, if this does not yet exists.
datapunt_processing.extract.download_from_ckan.download_metadata(url)

Download files from data catalog response id.

Args:
url: full data.amsterdam.nl url of the desired dataset, for example: https://data.amsterdam.nl/#?dte=catalogus%2Fapi%2F3%2Faction%2Fpackage_show%3Fid%3D5d84c216-b826-4406-8297-292678dee13c
Result:
All the Metadata from this dataset as a json dictonary, with the owner, refresh data, resource url’s to the desired files, etc.
datapunt_processing.extract.download_from_ckan.get_catalog_package_id(url)

Retrieve package id from full url from data.amsterdam.nl, for example: catalogus/api/3/action/package_show?id=c1f04a62-8b69-4775-ad83-ce2647a076ef

Args:
url: full data.amsterdam.nl url of the desired dataset, for example: https://data.amsterdam.nl/#?dte=catalogus%2Fapi%2F3%2Faction%2Fpackage_show%3Fid%3D5d84c216-b826-4406-8297-292678dee13c
Result:
Unique id number of package.
datapunt_processing.extract.download_from_ckan.parser()

Parser function to run arguments from commandline and to add description to sphinx.