Open Notificaties Documentation

Open Notificaties is a modern, open-source messaging service that implements the VNG API standard for the “Notificaties API” in line with the Common Ground model. Among others, it can be used in combination with Open Zaak which implements the VNG standards for “API’s voor Zaakgericht werken” that require a Notificaties API.

Getting Started

To get you started, you might find some of these links relevant:

  • New to Open Notificaties? Have a look at the Introduction

  • New to the VNG standards for “API’s voor Zaakgericht werken”? Read up on the API-specifications.

  • Want to get started with Open Notificaties yourself? See Installation.

  • Need help with Open Notificaties? Contact Support.

  • Are you a developer? Head over to Development (TODO)!

Open Notificaties is and only uses Open-source.

Introduction

Open Notificaties is a modern, open-source messaging service that implements the VNG API standard for the “Notificaties API” in line with the Common Ground model. Among others, it can be used in combination with Open Zaak which implements the VNG standards for “API’s voor Zaakgericht werken” that require a Notificaties API.

Open Notificaties exposes a Notificaties API (notifications) to route privacy-safe messages from publishers to subscribers.

Open Notificaties is based on the API reference implementations by VNG Realisatie to create a production-grade product that can be used by municipalities.

Architecture

Open Notificaties is based on the reference implementation of the “API’s voor Zaakgericht werken” made by VNG Realisatie. The overall architecture remains faithful to the Common Ground principles and all API specifications.

The architecture of Open Notificaties focusses on excellent performance, optimal stability and to guarantee data integrity. Under the hood, it uses RabbitMQ as message broker with just a light weight REST API on top of it.

It largely resembles the original reference implementation and remains its own component to scale separately from any other component.

Who’s behind Open Notificaties?

The Open Notificaties project was built for the Open Zaak project, initiated shortly before the 1.0 release of the VNG standards for “API’s voor Zaakgericht werken”, on August 1, 2019, by 9 municipalities:

  • Amsterdam

  • Rotterdam

  • Utrecht

  • Tilburg

  • Arnhem

  • Haarlem

  • ‘s-Hertogenbosch

  • Delft

  • Hoorn, Medemblik, Stede Broec, Drechteland, Enkhuizen (SED)

Using Dimpact as a legal entity, they formed a project to develop a modern, open-source data- and services-layer to enable zaakgericht werken, in line with the Common Ground model.

Open-source

Open Notificaties is open-source and available under the EUPL license.

In addition, this project makes use of various open-source Python libraries and npm packages under the hood.

API-specifications

Open Notificaties adheres to the API-specifications as described by the VNG API standard for the “Notificaties API”.

Supported API versions

The following API’s are available in Open Notificaties:

API

Specification version(s)

Notificaties API

1.0

Installation

There are several ways to install Open Zaak. A scalable solution is to use Kubernetes. You can also run the Docker containers on a single machine. Expert users can deploy everything on a linux machine directly.

Before you begin

  • Check the minimum system requirements for the target machine(s).

  • Make sure the target machine(s) have access to the Internet.

  • The target machine(s) should be reachable via at least a local DNS entry:

    • Open Zaak: open-zaak.<organization.local>

    • Open Notificaties: open-notificaties.<organization.local>

    The machine(s) do not need to be publically accessible and do not need a public DNS entry. In some cases, you might want this but it’s not recommended. The same machine can be used for both Open Zaak and Open Notificaties.

  • If you want to use NLX, make sure you have a publicaly available domain name, for example nlx.<organization.com>, where your NLX-inway is accessible to the outside world.

Hardware

Based on our initial performance tests in both a Kubernetes environment and single machine setups, we can indicate some minimum system requirements to reach a certain performance.

Notifications are likely to increase over time, due to more external components makes use of it, or more subscribers want to receive notifications. Hence, we recommend a Kubernetes setup since it will scale more easy.

Determine what you need

Since notifications are typically only sent for update and create actions, the number of notifications will most likely not be that high. Some functional views might cause a series of create and/or update actions, so there is a short burst of notifications which you will need to handle.

It really depends on how many consumers and other components will make use of Open Notificaties but assuming 50 requests per second will probably be a good start for any environment.

Minimum system requirements

  • Platform: 64-bit

  • Processor(s): 4 - 12 CPUs (see below) at 2.0 GHz

  • RAM: 4 - 12 GB (see below)

  • Hard disk space: 20 GB

Based on the number of requests per second you need, you can see what kind of hardware you need to achieve this.

Requests per second

CPUs

Memory (GB)

50

4

4

100

6

6

200

12

12

With these specifications you can run everything on a single machine or divided over several instances.

General recommendations

  • Use a seperate database server with roughly a third of the CPUs and memory as the main server. The database is usually the limiting factor.

Kubernetes recommendations

  • Preferably use 2 load balancer (like Traefik) replica’s.

  • Use as many replica’s as available CPU’s taking into account you need to have a few replica’s for your load balancer, and possibly other services.

Deploying on Kubernetes (TODO)
Deploying on a single server

Open Notificaties can be deployed on a single machine - either a dedicated server (DDS) or virtual private server (VPS). The required hardware can be rented from a hosting provider or be provided in your environment. Please see Hardware to determine the hardware requirements.

This documentation describes the architecture, prerequisites and how to deploy Open Notificaties on a server. Additionally, it documents the possible configuration options.

Note

The default settings allow Open Notificaties to be deployed to the same machine as Open Zaak.

Architecture

The application is deployed as Docker containers, of which the images are available on docker hub. Traffic is routed to the server, where the web server (nginx) handles SSL termination and proxies the requests to the application containers.

Data is stored in a PostgreSQL database. By default, the database is installed on the same machine (running on the host), but you can make use of a hosted database (Google Cloud, AWS, Azure…). See the Configuration parameters for more information.

Prerequisites

Before you can deploy, you need:

A server

Ensure you have a server with root privileges. We assume you can directly ssh to the machine as root user. If that’s not the case, a user with sudo will also work. Python 3 must be available on the server. Debian 9/10 are officially supported operating systems, though it is likely the installation also works on Ubuntu. CentOS/RedHat might work.

A copy of the deployment configuration

You can either clone the https://github.com/open-zaak/open-notificaties repository, or download and extract the latest ZIP: https://github.com/open-zaak/open-notificaties/archive/master.zip

Python and a Python virtualenv

You will need to have at least Python 3.5 installed on your system. In the examples, we assume you have Python 3.6.

Create a virtualenv with:

[user@laptop]$ python3.6 -m venv env/
[user@laptop]$ source env/bin/activate

Make sure to install the deployment tooling. In your virtualenv, install the dependencies:

(env) [user@laptop]$ pip install -r deployment/requirements.txt
(env) [user@laptop]$ ansible-galaxy collection install -r requirements.yml
(env) [user@laptop]$ ansible-galaxy role install -r requirements.yml
Deployment

Deployment is done with an Ansible playbook, performing the following steps:

  1. Install and configure PostgreSQL database server

  2. Install the Docker runtime

  3. Install the SSL certificate with Let’s Encrypt

  4. Setup Open Notificaties with Docker

  5. Install and configure nginx as reverse proxy

Initial steps

Make sure the virtualenv is activated:

[user@laptop]$ source env/bin/activate

Navigate to the correct deployment directory:

(env) [user@laptop]$ cd deployment/single-server

Create the vars/open-notificaties.yml file - you can find an example in vars/open-notificaties.yml.example. Generate a secret key using the django secret key generator and put the value between single quotes.

Configure the host by creating the hosts file from the example:

(env) [user@laptop]$ cp hosts.example hosts

Edit the open-notificaties.gemeente.nl to point to your actual domain name. You must make sure that the DNS entry for this domain points to the IP address of your server.

Warning

It’s important to use the correct domain name, as the SSL certificate will be generated for this domain and only this domain will be whitelisted by Open Notificaties! If you are using a private DNS name, then no SSL certificate can be created via Letsencrypt - make sure to disable it by setting certbot_create_if_missing=false.

Running the deployment

Execute the playbook by running:

(env) [user@laptop]$ ansible-playbook open-notificaties.yml

Hint

  • If you have your secrets Ansible vault encrypted, make sure you have either:

    • set the ANSIBLE_VAULT_PASSWORD_FILE environment variable, or

    • pass --ask-vault-pass flag to ansible-playbook.

  • If you need to override any deployment variables (see Configuration parameters), you can pass variables to ansible-playbook using the syntax: --extra-vars "some_var=some_value other_var=other_value".

  • If you want to run the deployment from the same machine as where it will run (ie. install to itself), you can pass --connection local to ansible-playbook.

  • If you cannot connect as root to the target machine, you can pass --user <user> --become --become-method=sudo --ask-become-pass which will connect as user <user> that needs sudo-rights on the target machine to install the requirements.

A full example might look like this:

(env) [user@laptop]$ ansible-playbook open-notificaties.yml \
    --user admin
    --inventory my-hosts \  # Use inventory file ``my-hosts`` instead of ``hosts``.
    --limit open-notificaties.gemeente.nl \  # Only pick open-notificaties.gemeente.nl from the inventory file.
    --extra-vars "certbot_create_if_missing=false app_db_name=opennotificaties-test app_db_user=opennotificaties-test" \
    --connection local \
    --become \
    --become-method=sudo \
    --ask-become-pass

Note

You can run the deployment multiple times, it will not affect the final outcome. If you decide to change configuration parameters, you do not have to start from scratch.

Environment configuration

After the initial deployment, some initial configuration is required. This configuration is stored in the database and is only needed once.

Create a superuser

A superuser allows you to perform all administrative tasks.

  1. Log in to the server:

    [user@laptop]$ ssh root@open-notificaties.gemeente.nl

  2. Create the superuser (interactive on the shell). Note that the password you type in will not be visible - not even with asterisks. This is normal.

    [root@open-notificaties.gemeente.nl]# docker exec -it opennotificaties-0 src/manage.py createsuperuser
Gebruikersnaam: demo
E-mailadres: admin@open-notificaties.gemeente.nl
Password:
Password (again):
Superuser created successfully.

Configure Open Notificaties Admin

  1. Open https://open-notificaties.gemeente.nl/admin/ in your favourite browser and log in with your superuser account.

  2. Navigate to Configuratie > Websites and edit example.com. Fill in your actual domain.

  3. Navigate to Configuratie > Notificatiescomponentconfiguratie and specify the correct Notificaties API url.

  4. Configure the credentials via API autorisaties.

Configuration parameters

At deployment time, you can configure a number of parts of the deployment by overriding variables. You can override variables on the command line (using the -e "..." syntax) or by overriding them in vars/secrets.yml.

Note

Tweaking configuration parameters is considered advanced usage.

Generic variables

  • certbot_admin_email: e-mail address to use to accept the Let’s Encrypt terms and conditions.

  • certbot_create_if_missing: whether to use Let’s Encrypt to create an SSL certificate for your domain. Set to false if you want to use an existing certificate.

Open Notificaties specific variables

The default values can be found in roles/opennotificaties/defaults/main.yml.

  • opennotificaties_db_port: database port. If you are running multiple PostgreSQL versions on the same machine, you’ll have to point to the correct port.

  • opennotificaties_db_host: specify the hostname if you’re using a cloud database or a database on a different server.

  • opennotificaties_db_name: specify a different database name.

  • opennotificaties_secret_key: A Django secret key. Used for cryptographic operations - this may NOT leak, ever. If it does leak, change it.

Scaling

The opennotificaties_replicas variable controls scaling on backend services. If your hardware allows it, you can create more replicas. By default, 3 replicas are running.

The format of each replica is:

name: opennotificaties-i
port: 900i

The port number must be available on the host - i.e. you may not have other services already listening on that port.

The opennotificaties_worker_replicas variable controls the scaling of the queue workers - these are responsible for actually distributing the notifications. By default, 3 replicas spin up.

The format of each replica is:

name: opennotificaties-worker-i
Updating an Open Notificaties installation

Make sure you have the deployment tooling installed - see the installation steps for more details.

If you have an existing environment (from the installation), make sure to update it:

# fetch the updates from Github
[user@host]$ git fetch origin

# checkout the tag of the version you wish to update to, e.g. 1.0.0
[user@host]$ git checkout X.Y.z

# activate the virtualenv
[user@host]$ source env/bin/activate

# ensure all (correct versions of the) dependencies are installed
(env) [user@host]$ pip install -r requirements.txt
(env) [user@host]$ ansible-galaxy install -r requirements.yml

Open Notificaties deployment code defines variables to specify the Docker image tag to use. This is synchronized with the git tag you’re checking out.

Next, to perform the upgrade, you run the open-notificaties.yml playbook just like with the installation in Running the deployment:

(env) [user@laptop]$ ansible-playbook open-notificaties.yml

Note

This will instruct the docker containers to restart using a new image. You may notice some brief downtime (order of seconds to minutes) while the new image is being downloaded and containers are being restarted.

Performance

Performance is measured using different types of scenarios.

Measuring performance
Goal

The purpose of performance measurements is to gain insight into the relationships between system requirements, number of users and response times of the API’s. From these relationships we can draw up the minimum system requirements, depending on the number of users, on which the API’s still perform acceptable.

A standardized performance measurement also provides insight into which effect various optimizations have.

Technical test scenarios

There’s basically just one API-call that will be called often and that’s to send a notification. The other API-calls are neglible. The API-call (request) is measured to gain insights in the technical performance.

Notificaties API

  1. Create NOTIFICATIE (POST /api/v1/notificaties)

Test specification
Using scenarios

A scenario in this test specification is equal to an API-call (request). Each API resource is continuously without any delay, or waiting time, between each request. This way, we can determine the maximum number of requests per second and average response times.

Virtual users

We test with an increasing number of virtual users, from 1 to 100, that concurrently execute the test scenarios. A virtual user is technically a script that executes the different scenarios one after another. This way, we see the number of virtual users that concurrently access the API’s and its impact on performance.

Testdata

There’s not test data involved.

Support

Development is done separately from any support or service that you might need to install, use, implement or integrate Open Notificaties.

What kind of support do you need?

  • Need help with the installation? Contact a service provider!

  • Found a bug? Please create an issue on Github!

  • Want to contribute? See Development (TODO)!

Service providers

Below is a list of service providers.

Want to be mentioned here? Please create a Pull Request on Github to add your company.

Development (TODO)

Open Notificaties is open-source software. We’d love to have you contribute!

Contributing

Development environment