Open Notificaties Documentation¶
Open Notificaties is a modern, open-source messaging service that implements the VNG API standard for the “Notificaties API” in line with the Common Ground model. Among others, it can be used in combination with Open Zaak which implements the VNG standards for “API’s voor Zaakgericht werken” that require a Notificaties API.
Getting Started¶
To get you started, you might find some of these links relevant:
New to Open Notificaties? Have a look at the Introduction
New to the VNG standards for “API’s voor Zaakgericht werken”? Read up on the API-specifications.
Want to get started with Open Notificaties yourself? See Installation.
Need help with Open Notificaties? Contact Support.
Are you a developer? Head over to Development (TODO)!
Open Notificaties is and only uses Open-source.
Introduction¶
Open Notificaties is a modern, open-source messaging service that implements the VNG API standard for the “Notificaties API” in line with the Common Ground model. Among others, it can be used in combination with Open Zaak which implements the VNG standards for “API’s voor Zaakgericht werken” that require a Notificaties API.
Open Notificaties exposes a Notificaties API (notifications) to route privacy-safe messages from publishers to subscribers.
Open Notificaties is based on the API reference implementations by VNG Realisatie to create a production-grade product that can be used by municipalities.
Architecture¶
Open Notificaties is based on the reference implementation of the “API’s voor Zaakgericht werken” made by VNG Realisatie. The overall architecture remains faithful to the Common Ground principles and all API specifications.
The architecture of Open Notificaties focusses on excellent performance, optimal stability and to guarantee data integrity. Under the hood, it uses RabbitMQ as message broker with just a light weight REST API on top of it.
It largely resembles the original reference implementation and remains its own component to scale separately from any other component.
Who’s behind Open Notificaties?¶
The Open Notificaties project was built for the Open Zaak project, initiated shortly before the 1.0 release of the VNG standards for “API’s voor Zaakgericht werken”, on August 1, 2019, by 9 municipalities:
Amsterdam
Rotterdam
Utrecht
Tilburg
Arnhem
Haarlem
‘s-Hertogenbosch
Delft
Hoorn, Medemblik, Stede Broec, Drechteland, Enkhuizen (SED)
Using Dimpact as a legal entity, they formed a project to develop a modern, open-source data- and services-layer to enable zaakgericht werken, in line with the Common Ground model.
Open-source¶
Open Notificaties is open-source and available under the EUPL license.
In addition, this project makes use of various open-source Python libraries and npm packages under the hood.
API-specifications¶
Open Notificaties adheres to the API-specifications as described by the VNG API standard for the “Notificaties API”.
Supported API versions¶
The following API’s are available in Open Notificaties:
API |
Specification version(s) |
---|---|
Installation¶
There are several ways to install Open Notificaties. A scalable solution is to use Kubernetes. You can also run the Docker containers on a single machine
Before you begin¶
Check the minimum system requirements for the target machine(s).
Make sure the target machine(s) have access to the Internet.
The target machine(s) should be reachable via at least a local DNS entry:
Open Zaak:
open-zaak.<organization.local>
Open Notificaties:
open-notificaties.<organization.local>
The machine(s) do not need to be publically accessible and do not need a public DNS entry. In some cases, you might want this but it’s not recommended. The same machine can be used for both Open Zaak and Open Notificaties.
If you want to use NLX, make sure you have a publicaly available domain name, for example
nlx.<organization.com>
, where your NLX-inway is accessible to the outside world.
Guides¶
Hardware¶
Based on our initial performance tests in both a Kubernetes environment and single machine setups, we can indicate some minimum system requirements to reach a certain performance.
Notifications are likely to increase over time, due to more external components makes use of it, or more subscribers want to receive notifications. Hence, we recommend a Kubernetes setup since it will scale more easy.
Determine what you need¶
Since notifications are typically only sent for update and create actions, the number of notifications will most likely not be that high. Some functional views might cause a series of create and/or update actions, so there is a short burst of notifications which you will need to handle.
It really depends on how many consumers and other components will make use of Open Notificaties but assuming 50 requests per second will probably be a good start for any environment.
Minimum system requirements¶
Platform: 64-bit
Processor(s): 4 - 12 CPUs (see below) at 2.0 GHz
RAM: 4 - 12 GB (see below)
Hard disk space: 20 GB
Based on the number of requests per second you need, you can see what kind of hardware you need to achieve this.
Requests per second |
CPUs |
Memory (GB) |
50 |
4 |
4 |
100 |
6 |
6 |
200 |
12 |
12 |
With these specifications you can run everything on a single machine or divided over several instances.
Use a seperate database server with roughly a third of the CPUs and memory as the main server. The database is usually the limiting factor.
Preferably use 2 load balancer (like Traefik) replica’s.
Use as many replica’s as available CPU’s taking into account you need to have a few replica’s for your load balancer, and possibly other services.
Deploying on Kubernetes (TODO)¶
Deploying on a single server¶
Open Notificaties can be deployed on a single machine - either a dedicated server (DDS) or virtual private server (VPS). The required hardware can be rented from a hosting provider or be provided in your environment. Please see Hardware to determine the hardware requirements.
This documentation describes the architecture, prerequisites and how to deploy Open Notificaties on a server. Additionally, it documents the possible configuration options.
Note
The default settings allow Open Notificaties to be deployed to the same machine as Open Zaak.
Architecture¶
The application is deployed as Docker containers, of which the images are available on docker hub. Traffic is routed to the server, where the web server (nginx) handles SSL termination and proxies the requests to the application containers.
Data is stored in a PostgreSQL database. By default, the database is installed on the same machine (running on the host), but you can make use of a hosted database (Google Cloud, AWS, Azure…). See the Configuration parameters for more information.
Prerequisites¶
Before you can deploy, you need:
Ensure you have a server with root
privileges. We assume you can directly
ssh to the machine as root
user. If that’s not the case, a user with
sudo
will also work. Python 3 must be available on the server. Debian 9/10
are officially supported operating systems, though it is likely the
installation also works on Ubuntu. CentOS/RedHat might work.
You can either clone the https://github.com/open-zaak/open-notificaties repository, or download and extract the latest ZIP: https://github.com/open-zaak/open-notificaties/archive/master.zip
You will need to have at least Python 3.5 installed on your system. In the examples, we assume you have Python 3.6.
Create a virtualenv with:
[user@laptop]$ python3.6 -m venv env/
[user@laptop]$ source env/bin/activate
Make sure to install the deployment tooling. In your virtualenv, install the dependencies:
(env) [user@laptop]$ pip install -r deployment/requirements.txt
(env) [user@laptop]$ ansible-galaxy collection install -r requirements.yml
(env) [user@laptop]$ ansible-galaxy role install -r requirements.yml
Deployment¶
Deployment is done with an Ansible playbook, performing the following steps:
Install and configure PostgreSQL database server
Install the Docker runtime
Install the SSL certificate with Let’s Encrypt
Setup Open Notificaties with Docker
Install and configure nginx as reverse proxy
Make sure the virtualenv is activated:
[user@laptop]$ source env/bin/activate
Navigate to the correct deployment directory:
(env) [user@laptop]$ cd deployment/single-server
Create the vars/open-notificaties.yml
file - you can find an example in
vars/open-notificaties.yml.example
. Generate a secret key using the
django secret key generator and put the value between single
quotes.
Configure the host by creating the hosts
file from the example:
(env) [user@laptop]$ cp hosts.example hosts
Edit the open-notificaties.gemeente.nl
to point to your actual domain name. You must
make sure that the DNS entry for this domain points to the IP address of your
server.
Warning
It’s important to use the correct domain name, as the SSL certificate
will be generated for this domain and only this domain will be whitelisted
by Open Notificaties! If you are using a private DNS name, then no SSL certificate
can be created via Letsencrypt - make sure to disable it by setting
certbot_create_if_missing=false
.
Execute the playbook by running:
(env) [user@laptop]$ ansible-playbook open-notificaties.yml
Hint
If you have your secrets Ansible vault encrypted, make sure you have either:
set the
ANSIBLE_VAULT_PASSWORD_FILE
environment variable, orpass
--ask-vault-pass
flag toansible-playbook
.
If you need to override any deployment variables (see Configuration parameters), you can pass variables to
ansible-playbook
using the syntax:--extra-vars "some_var=some_value other_var=other_value"
.If you want to run the deployment from the same machine as where it will run (ie. install to itself), you can pass
--connection local
toansible-playbook
.If you cannot connect as
root
to the target machine, you can pass--user <user> --become --become-method=sudo --ask-become-pass
which will connect as user<user>
that needssudo
-rights on the target machine to install the requirements.
A full example might look like this:
(env) [user@laptop]$ ansible-playbook open-notificaties.yml \
--user admin
--inventory my-hosts \ # Use inventory file ``my-hosts`` instead of ``hosts``.
--limit open-notificaties.gemeente.nl \ # Only pick open-notificaties.gemeente.nl from the inventory file.
--extra-vars "certbot_create_if_missing=false app_db_name=opennotificaties-test app_db_user=opennotificaties-test" \
--connection local \
--become \
--become-method=sudo \
--ask-become-pass
Note
You can run the deployment multiple times, it will not affect the final outcome. If you decide to change configuration parameters, you do not have to start from scratch.
After the initial deployment, some initial configuration is required. This configuration is stored in the database and is only needed once.
Create a superuser
A superuser allows you to perform all administrative tasks.
Log in to the server:
[user@laptop]$ ssh root@open-notificaties.gemeente.nl
Create the superuser (interactive on the shell). Note that the password you type in will not be visible - not even with asterisks. This is normal.
[root@open-notificaties.gemeente.nl]# docker exec -it opennotificaties-0 src/manage.py createsuperuser Gebruikersnaam: demo E-mailadres: admin@open-notificaties.gemeente.nl Password: Password (again): Superuser created successfully.
Configure Open Notificaties Admin
See the Open Notificaties configuration on how to configure Open Notificaties post-installation.
Configuration parameters¶
At deployment time, you can configure a number of parts of the deployment by
overriding variables. You can override variables on the command line (using the
-e "..."
syntax) or by overriding them in vars/secrets.yml
.
Note
Tweaking configuration parameters is considered advanced usage.
certbot_admin_email
: e-mail address to use to accept the Let’s Encrypt terms and conditions.certbot_create_if_missing
: whether to use Let’s Encrypt to create an SSL certificate for your domain. Set tofalse
if you want to use an existing certificate.
The default values can be found in roles/opennotificaties/defaults/main.yml
.
opennotificaties_db_port
: database port. If you are running multiple PostgreSQL versions on the same machine, you’ll have to point to the correct port.opennotificaties_db_host
: specify the hostname if you’re using a cloud database or a database on a different server.opennotificaties_db_name
: specify a different database name.opennotificaties_secret_key
: A Django secret key. Used for cryptographic operations - this may NOT leak, ever. If it does leak, change it.
Scaling
The opennotificaties_replicas
variable controls scaling on backend services. If
your hardware allows it, you can create more replicas. By default, 3 replicas
are running.
The format of each replica is:
name: opennotificaties-i
port: 900i
The port number must be available on the host - i.e. you may not have other services already listening on that port.
The opennotificaties_worker_replicas
variable controls the scaling of the queue
workers - these are responsible for actually distributing the notifications. By default,
3 replicas spin up.
The format of each replica is:
name: opennotificaties-worker-i
Updating an Open Notificaties installation¶
Make sure you have the deployment tooling installed - see the installation steps for more details.
If you have an existing environment (from the installation), make sure to update it:
# fetch the updates from Github
[user@host]$ git fetch origin
# checkout the tag of the version you wish to update to, e.g. 1.0.0
[user@host]$ git checkout X.Y.z
# activate the virtualenv
[user@host]$ source env/bin/activate
# ensure all (correct versions of the) dependencies are installed
(env) [user@host]$ pip install -r requirements.txt
(env) [user@host]$ ansible-galaxy install -r requirements.yml
Open Notificaties deployment code defines variables to specify the Docker image tag to use. This is synchronized with the git tag you’re checking out.
Next, to perform the upgrade, you run the open-notificaties.yml
playbook just like
with the installation in Running the deployment:
(env) [user@laptop]$ ansible-playbook open-notificaties.yml
Note
This will instruct the docker containers to restart using a new image. You may notice some brief downtime (order of seconds to minutes) while the new image is being downloaded and containers are being restarted.
Open Notificaties configuration¶
Before you can work with Open Notificaties after installation, a few settings need to be configured first.
Configure Notificaties API¶
Open Notificaties uses the Autorisaties API to check if the sender is authorized to send notifications. Open Zaak offers an Autorisaties API and below we assume you use this one.
Configure the Open Zaak Autorisaties API endpoint (so Open Notificaties knows where to check for the proper autorisations):
Navigate to Configuratie > Autorisatiecomponentconfiguratie
Fill out the form:
API root: The URL to the Notificaties API. For example:
https://open-zaak.gemeente.local/autorisaties/api/v1/
.Component:
Notificatierouteringcomponent
Click Opslaan.
For a complete example of setting up Open Zaak to send notifications to Open Notificaties, see the documentation of Open Zaak.
Below are the general steps to allow an application to send notifications and using Open Zaak as the example.
Configure the credentials for the Open Zaak Autorisaties API (so Open Notificaties can access the Autorisaties API):
Navigate to API Autorisaties > Externe API credentials
Click Externe API credential toevoegen.
Fill out the form:
API root: The URL of the Open Zaak Autorisaties API endpoint
Label: For example:
Open Zaak Autorisaties
Client ID: For example:
open-notificaties
Secret: Some random string
User ID: Same as the Client ID
User representation: For example:
Open Notificaties
Make sure Open Notificaties is authorized in Open Zaak to access the Autorisaties API by using the same Client ID and Secret as given here.
Click Opslaan.
Finally, we need to allow Open Zaak to access Open Notificaties (for authentication purposes, so we can then check its authorizations):
Navigate to API Autorisaties > Client credentials
Click Client credential toevoegen.
Fill out the form:
Client ID: The same Client ID as given in Open Zaak step 2c
Secret: The same Secret as given in Open Zaak step 2c
Make sure Open Zaak is configured to use this Client ID and secret to access Open Notificaties.
Click Opslaan.
All done!
Post-install checklist¶
After Open Notificaties has been installed successfully, go through the following checklist to see if the software works as expected:
Check configuration
Check the configuration page for Open Notificaties, accessible at the url
https://open-notificaties.gemeente.nl/view-config/
.
This page will indicate whether certain settings are properly configured.
Run check management commands
If Sentry was set up for Open Notificaties, make sure to run the following command to ensure that logging to Sentry will work as expected:
python src/manage.py raven test
Manual¶
Note
The manual is currently only available in the Dutch language!
Once Open Notificaties is installed, it can be further configured via the management interface.
ADFS (On premise) Configureren¶
Open Notificaties ondersteunt Single Sign On (SSO) via ADFS voor de beheerinterface.
Gebruikers kunnen op die manier inloggen op Open Notificaties met hun ADFS account. In deze flow:
Klikt een gebruiker op het inlogscherm op Inloggen met ADFS
De gebruiker wordt naar de ADFS omgeving geleid waar ze inloggen met gebruikersnaam en wachtwoord (en eventuele Multi Factor Authorization)
De ADFS omgeving stuurt de gebruiker terug naar Open Notificaties (waar de account aangemaakt wordt indien die nog niet bestaat)
Een beheerder in Open Notificaties kent de juiste groepen toe aan deze gebruiker als deze voor het eerst inlogt.
Note
Standaard krijgen deze gebruikers geen toegang tot de beheerinterface. Deze rechten moeten door een (andere) beheerder ingesteld worden. De account is wel aangemaakt indien deze nog niet bestond.
Configureren van ADFS zelf¶
Contacteer de IAM/ADFS-beheerders in je organisatie om een Application Group aan te maken in de ADFS omgeving. Er zijn configuratiehandleidingen (Engels) beschikbaar.
Voor de Redirect URI vul je https://open-notificaties.gemeente.nl/adfs/callback
in, waarbij je open-notificaties.gemeente.nl
vervangt door het relevante domein.
Aan het eind van dit proces moet je de volgende gegevens hebben (on premise):
Server adres, bijvoorbeeld
login.gemeente.nl
Client ID, bijvoorbeeld
3ae1852d-bf76-4731-9c41-1a31928cf6a6
Configureren van ADFS in Open Notificaties¶
Zorg dat je de volgende gegevens hebt:
Server adres
Client ID
Navigeer vervolgens in de admin naar Configuratie > ADFS Confiugration.
Vink Enable aan om ADFS in te schakelen.
Vul bij Server (on premise) het server adres in, bijvoorbeeld
login.gemeente.nl
.Vul bij Client ID het Client ID in, bijvoorbeeld
3ae1852d-bf76-4731-9c41-1a31928cf6a6
.Vul bij Relying Party ID opnieuw het Client ID in, bijvoorbeeld
3ae1852d-bf76-4731-9c41-1a31928cf6a6
.Laat bij Claim mapping de standaardwaarden staan.
Vul bij Username claim de waarde
winaccountname
in.
Klik tot slot rechtsonder op Opslaan.
Je kan vervolgens het makkelijkst testen of alles werkt door in een incognitoscherm naar https://open-notificaties.gemeente.nl/admin/ te navigeren en op Inloggen met ADFS te klikken.
Support¶
Development is done separately from any support or service that you might need to install, use, implement or integrate Open Notificaties.
What kind of support do you need?
Need help with the installation? Contact a service provider!
Found a bug? Please create an issue on Github!
Want to contribute? See Development (TODO)!
Development (TODO)¶
Open Notificaties is open-source software. We’d love to have you contribute!
Contributing¶
Development environment¶
Performance¶
Performance is measured using different types of scenarios.
Measuring performance¶
Goal¶
The purpose of performance measurements is to gain insight into the relationships between system requirements, number of users and response times of the API’s. From these relationships we can draw up the minimum system requirements, depending on the number of users, on which the API’s still perform acceptable.
A standardized performance measurement also provides insight into which effect various optimizations have.
Technical test scenarios¶
There’s basically just one API-call that will be called often and that’s to send a notification. The other API-calls are neglible. The API-call (request) is measured to gain insights in the technical performance.
Notificaties API
Create NOTIFICATIE (
POST /api/v1/notificaties
)
A scenario in this test specification is equal to an API-call (request). Each API resource is continuously without any delay, or waiting time, between each request. This way, we can determine the maximum number of requests per second and average response times.
We test with an increasing number of virtual users, from 1 to 100, that concurrently execute the test scenarios. A virtual user is technically a script that executes the different scenarios one after another. This way, we see the number of virtual users that concurrently access the API’s and its impact on performance.
There’s not test data involved.
Changes¶
1.1.0 (2020-03-16)¶
Feature and small improvements release.
Note
The API remains unchanged.
Removed unnecessary sections in documentation
Updated deployment examples
Tweak deployment to not conflict (or at least less likely :-) ) with Open Zaak install Open Zaak and Open Notificaties on the same machine are definitely supported
Added support for ADFS Single Sign On (disabled by default)
Added documentation build to CI
1.0.0 final (2020-02-07)¶
🎉 First stable release of Open Notificaties.
Features:
Notificaties API implementation
Tested with Open Zaak integration
Admin interface to view data created via the APIs
Scalable notification delivery workers
NLX ready (can be used with NLX)
Documentation on https://open-notificaties.readthedocs.io/
Deployable on Kubernetes, single server and as VMware appliance
Automated test suite
Automated deployment