Python backend projects
This document describes the common practices that are used in our Python backend projects. It is not meant as an encompassing development standards document. Within the backend team developers are free to choose the best tool for the job.
How we use docker and docker-compose
A lot of the open source code software of the City of Amsterdam uses Docker containers. Separate tasks are normally organised in separate Dockers.
In general we follow common community guidelines:
- Dockerfile best practices , including multi-stage builds
Preferably use docker-ce and docker-compose, most recent version
.dockerignore files are used to exclude files that are not relevant to the build.
Ideally this file starts by ignoring everything (
*) and then explicitly add the files that belong to the build
by means of exceptions (e.g.
!**\*.py to include all python source files).
Most often you will however find .dockerignore files that only specify files that are not part of the build.
How we use Python
See Python Style Guide
What you can expect to find in our project structure
each defined in a separate folder at the root level of the project
each having separate sources and tests folders
each having a
Dockerfilewith a corresponding
each having a
requirements.txtfile to install the package
- jenkins or similar ci yaml to configure the continuous build and integration
How we test
All of our projects require (unit) tests which are integrated in the continuous build process.
For testing we prefer using pytest, following its corresponding good practices
However you may find the use of unittest in some (older) projects.
Code coverage is normally not checked. Absolute code coverage requirements and enforcement thereof exist only in very rare cases.
How we assure our code quality
All code is reviewed by a colleague and subsequently accepted in the acceptance environment, before it is released to the production environment.
Deployment to production is a manual action in the Continuous Integration (CI) environment.
Linting tools (pylint, flake8, pycodestyle, …) and the inclusion of these checks in the CI cycle are rarely done in our projects. Most of the linting is expected to be done by the IDE. Code quality checks are normally not part of the build process.CI
How we build our API’s
Our API’s are documented using swagger. Each project is able to serve a swagger json or yaml file. The endpoint is included in the catalog project and so exposing it to our clients.
Swagger first is preferred. This means that the swagger specification determines the API service.
Common practice is however to start with the development of the API and generate or write the swagger documentation afterwards.
Which frameworks we use
Django and Flask are used alternately. Projects that are tightly connected to databases normally use the Django framework. Other (mostly smaller) projects often make use of the Flask framework.
In recent projects the aiohttp asynchronous HTTP client/server is used.
Recently we also use [golang](https://golang.org/). The specific practices for these projects will be described in a seperate guide.
How we share code between our projects
We are in the process of packaging our shared code. Links to relevant packages will be included in this guide when they become available.
However, you will also find shared code that has been included in other projects by simply copying its contents.
A real template project that is used to bootstrap a new project does not exist within our repository. There exists however a project that serves more or less as our showcase project: Monumenten. The setup and layout from this project can be taken as a starting point for new projects.
Which DataBase Management System (DBMS) we use
PostgreSQL. In Flask projects the SQLAlchemy library is used to access the database.
How we use github
Issues are registered in Jira or Taiga. Git issues are not used. Other github users will normally not be able to check for any solved or open issues. Neither is any procedure in place to respond to new issues that are registerd in github.
Normally not used, all code resides in one branch; the master branch.
Pull requests are rarely used. Updates are normally directly pushed onto the master branch.
Branches like master or development are not protected against direct updates.
Reviews are on the basis of commits. A review task is taken from Jira or Taiga. In the review task the reference to the commit is included. The commit description and the description of the original task is taken as input for the review. There is normally no description of how the modification can be tested.
Merge or rebase
Common practice is that commits are rebased. Rebasing not only keeps the master history clean, it also ensures that each commit in the master history has been tested, reviewed and works.