Modules¶
Data Processing contains 4 main modules:
- Extract
- Transform
- Load
- Helpers
Basic usage¶
Fill in the config.ini with the proper testuser credentials for the project. Do not store passwords in this file but use ENV variables with `export ENV=****` Use .gitignore to ignore .ini files to prevent uploading them to github.
Extract¶
Functions to extract data from the Objectstore where raw files are stored, from API’s or from the data catalog.
- download_from_catalog
- download_from_objectstore
- download_from_wfs
- download_from_api_with_authentication
- download_from_api_brk
- download_from_api_tellus
- download_from_api_kvk
- download_from_ckan
- download_tables_from_dokuwiki_to_json
- download_from_signals_api
- download_all_resources_from_dcatd_to_csv
- write_table_to_csv
- write_table_to_geojson
- write_xml_to_df_to_csv
- write_xml_to_json
- write_mdb_to_csv
- write_csv_to_dataframe
Transform Geospatial¶
Functions to spatial transform and enrich datasets.
Load¶
Common functions to load data into the Objectstore, CKAN (our public data.amsterdam.nl) or PostgreSQL.
Helpers¶
Generic helper functions for most commonly used data operations.