How we set up logging and monitoring


At Datapunt new applications, websites and data sources are released every month. To be able to monitor the use of these services, and to see the performance of our products, we use a standardised format for logging. To ensure scalability, up time monitoring, performance monitoring and error monitoring, some basic data needs to be logged by all applications. This guide contains instructions on how logging is implemented in our software.

For technical details about logging and monitoring in general, including notes on frontend- and backend logging, reporting and dashboarding, Amsterdam officials can view the general logging and monitoring dokuwiki.

Frontend logging

All user statistics are logged in Matomo, formerly named Piwik. This web application monitors visits, unique visitors, visited pages, their urls, downloads, bounce rates, et cetera. It is fairly easy to implement these basic measurements by placing a small piece of Javascript on your site. You can find instructions here.

There are also possibilities to create custom events for every button click or action that’s available on your website, set goals and measure their conversions, create custom variables and dimensions, manage privacy and security and more. Some excellent middleware has been developed for these functionalities which can be found in this repo.

The log files are kept on our own servers and can be called either through Matomo’s API or visualised in Matomo’s dashboarding tool.

Further technical details and implementation choices specifically about can be viewed by Amsterdam officials on the sata and information dokuwiki.

Backend logging

On an application level every API call is logged in JSON format. The technical details can be found on the general logging and monitoring dokuwiki.

All the logs are then picked up by logstash/filebeat/elasticsearch. They can be inspected, aggregated and visualised in Kibana. Standard visualisations and dashboards are available on request from team Service Delivery.

Error monitoring

Error monitoring can be done through the Kibana interface. Some developers also add a Sentry config to get notified when and where errors are occuring.

Up time monitoring

We use Up Time Robot to check if our applications are up and running. Adding an empty file to YOUR_URL/status/health on your site allows team Basis to add a health check for your site to the uptimerobot config.

All Guides


How to contribute to this City of Amsterdam Open Source project

Python backend projects

The style guide to the way we organize our Python back-end projects

How to code for humans

What we should think of when writing code so the most important computer we work with—the human brain—can parse it effectively

How we create cleaned, reproducable data for use in projects and apps

The way we make reusable data etl pipelines

How we create a docker environment for data analysis

How we set up a docker environment for analysis

How we set up logging and monitoring

How to incorporate logging to your applications and how to visualize this data

How we code Python

The style guide to the way we code Python

How we track Open Source health

Understanding the impact and health of Open Source projects

How to write a README

The goto file to see what a project is and how to use it