download_from_catalog¶
Get data and metadata from data.amsterdam.nl, unzip if needed and put the file in a local directory. To test run this command line:download_from_catalog https://data.amsterdam.nl/#?dte=dcatd%2Fdatasets%2Finzameldagen-grofvuil-huisvuil&dtfs=T&mpb=topografie&mpz=11&mpv=52.3731081:4.8932945 data
usage: download_from_data_amsterdam_catalog [-h] url output_folder
Positional Arguments¶
| url |
|
| output_folder |
|
functions¶
-
datapunt_processing.extract.download_from_catalog.download_all_files(metadata, download_directory)¶ Download all files from metadata resources list.
- Args:
- metadata: json dictonary from ckan with all the metadata including the resources list of all files.
- download_directory: path where to store the files from the files, for example data.
- Result:
- Unzipped and created dir filled with all data in the download_directory, if this does not yet exists.
-
datapunt_processing.extract.download_from_catalog.download_metadata(url)¶ Download files from data catalog using the dcatd identifier.
- Args:
- url: full data.amsterdam.nl url of the desired dataset, for example: https://data.amsterdam.nl/#?dte=dcatd%2Fdatasets%2Finzameldagen-grofvuil-huisvuil&dtfs=T&mpb=topografie&mpz=11&mpv=52.3731081:4.8932945
- Result:
- All the Metadata from this dataset as a json dictonary, with the owner, refresh data, resource url’s to the desired files, etc.
-
datapunt_processing.extract.download_from_catalog.get_catalog_package_id(url)¶ Retrieve dcatd URI from full url from data.amsterdam.nl, for example: dcatd/datasets/inzameldagen-grofvuil-huisvuil
- Args:
- url: full data.amsterdam.nl url of the desired dataset, for example: https://data.amsterdam.nl/#?dte=dcatd%2Fdatasets%2Finzameldagen-grofvuil-huisvuil&dtfs=T&mpb=topografie&mpz=11&mpv=52.3731081:4.8932945
- Result:
- Unique id number of package.
-
datapunt_processing.extract.download_from_catalog.parser()¶ Parser function to run arguments from commandline and to add description to sphinx.