This is a series of blog posts about the Message Room concept. The first part provides an introduction to synchronous and asynchronous data exchange. The second part concentrates on life-event-based services and potential implementation alternatives. Three alternative implementation approaches are discussed in more detail in the third part.

In my previous blog post, I covered five different ways how the Message Room concept could be implemented. In this blog post, I will explain in more detail how NIIS has approached three of the five alternatives at a more practical level. During the last two years, NIIS has conducted multiple research and development activities that have concentrated on alternatives 1-3 (built-in, integrated, connected).

Integrating Apache Kafka into X-Road (integrated)

In collaboration with the University of Tartu, NIIS conducted a research project on supporting event-driven architecture in the context of X-Road. The study consisted of two parts that were completed in 2020-2021. The final report is available in the X-Road Document Library.

The first part of the project produced a report (requirement analysis and feasibility study) and a proof-of-concept implementation which brought a subset of Apache Kafka's capabilities into X-Road. The second part studied Apache Kafka's integration into X-Road further and explored topics that must be covered in a production level integration, for example, high availability, authentication, access rights management, and service discovery. Implementing Kafka management operations in X-Road was out of the project’s scope.

Image 1. Integrating Apache Kafka into X-Road (integrated).

The integration is based on the idea that the existing X-Road protocols are used for a handshake to establish an asynchronous communication channel between the message exchange parties. The asynchronous communication is implemented by the new XGate add-on developed in the project. First, the service consumer sends a regular X-Road request to the service provider using the X-Road Message Protocol for REST (1.). The request is transmitted between the Security Servers using the X-Road Message Transport Protocol. The service provider receives the message, instantiates an interface for asynchronous communication, and adds routing information in the response (2.). The service consumer reads the routing information from the response, establishes a connection to the interface (3.), and reads data from it (4.). More detailed description of the implementation is available in the final report.

The adapter service approach (connected)

X-Road-Kafka Adapter is an adapter component that connects Apache Kafka topics to X-Road. The Adapter supports both producing and consuming data over a simple REST API. The producers publish data to a topic, and the consumers poll the topic and pull data from it. The Adapter sits between the Security Server and Apache Kafka and converts messages between the X-Road message protocol for REST and Kafka protocol. However, the Adapter doesn’t support streaming.

In practice, both producers and consumers may use Kafka's native API directly, or alternatively, they can use a REST API provided by the Adapter over X-Road. In real life, producers and consumers owned by the organisation running Kafka would probably use the native API, whereas external producers and consumers would connect to Kafka through X-Road. However, all the Kafka management and maintenance operations must be done using the native API since the Adapter only supports a limited subset of operations for producing and consuming messages.

More detailed information about the Adapter is available on GitHub. The Adapter is currently on a proof-of-concept level and requires further development before it can be used to run production workloads. The source code is licensed under the MIT open-source license and anyone interested in the Adapter is welcome to contribute to its development.

The X-Road way (built-in)

NIIS is currently working on a proof-of-concept (PoC) level implementation of the built-in approach. The goal of the PoC is to implement the Message Room concept as a Security Server proxy add-on (like messagelog, metaservice, op-monitoring, etc.). In that way, the implementation is modular and can be installed on selected Security Servers only. However, the add-on needs to be installed only on Security Servers that are acting as publishers. No code changes are required on Security Servers acting as subscribers, which means that all existing supported Security Server versions can act as subscribers.

It must be noted that the features included in the PoC are just a narrow subset of potential features of a production-level implementation. The PoC aims to test the concept with minimum viable features, and therefore, many features and functionalities are left out on purpose. The PoC implementation covers the following functionality:

Push-push publish/subscribe model.
- Publishers push messages to a Message Room.
- A Message Room pushes messages to subscribers - subscribers don't need to poll the Message Room.
One publisher per Message Room.
One publisher Security Server per Message Room.
Multiple subscribers per Message Room.
Message Rooms are public - anyone can subscribe to them.
Security Server provides the required interfaces to:
- Publish messages to a Message Room.
- Subscribe to a Message Room.
- Unsubscribe from a Message Room.
Support for federation.
Message Rooms are content-type and payload agnostic.
- Messages that are published to a Message Room can be of any content-type, e.g., XML, JSON, text, binary, etc.
All X-Road security guarantees (except access control for Message Rooms) are supported.

The following restrictions apply to the PoC implementation:

Pulling data from a Message Room is not supported.
- In push-pull model publishers push messages to a Message Room, and subscribers pull the messages from the Message Room.
Private Message Rooms are not supported.
- It's not possible to control who's allowed to subscribe to a Message Room.
Internal load-balancing is not supported.
- It's not possible to publish messages to a Message Room from multiple Security Servers.
- A Message Room is coupled with a single Security Server.
No error handling.
- If the recipient is not available, the message is lost.
- If publisher's Security Server crashes, (some) messages are lost.
No support for service discovery.
- There's no automated way to discover what Message Rooms are available.
- Potential subscribers must know the subsystem code of the Message Room when subscribing to it.
No changes to the Security Server UI and management REST API.
- If new configuration items are introduced, they're values are configured using configuration files or database queries.
Subscribe and unsubscribe interfaces support only REST. SOAP is not supported.

The Message Room add-on

The Message Room PoC is implemented as a Security Server add-on.

The add-on includes three new interfaces:

Publish - publish messages to a Message Room.
Subscribe - subscribe to a Message Room.
Unsubscribe - unsubscribe from a Message Room.

The publish interface is used by internal clients to publish messages to a Message Room. The publisher of the message is defined using the "X-Road-Client" HTTP header. The message body is fully data format-agnostic, just like the REST interface. The Message Room where the message is published is the same as the client subsystem. However, only subsystems with a special Message Room status can be used as a Message Room.

The subscribe and unsubscribe interfaces are used by external clients to manage their subscriptions to a Message Room. The subscribe interface is used to subscribe to a Message Room. Similarly, the unsubscribe interface is used to unsubscribe from a Message Room. In the PoC implementation, the interfaces are accessible by anyone, and there's no access control to them. For example, listMethods and listAllowedMethods metaservices work in the same way.

The message body of the subscribe interface must contain the service ID where messages published to the Message Room are sent. Also, the service must be owned by the same client who sends the subscribe message. The same applies to unsubscribe requests, too - the sender of the request must be the owner of the service specified in the request body. Also, subscribers must allow the publisher to send messages to the service specified in the subscribe message.

Image 4. The message flow using the built-in approach.

The basic message flow for publishing messages is explained in the diagram above:

A publisher publishes a message to a Message Room using the publish endpoint.
The proxy stores the message in memory and returns an acknowledgment message to the publisher.
A scheduled Message Room Processor reads new unprocessed messages from memory and reads the subscriber service IDs from the serverconf database.
The Message Room Processor sends the message to the subscribers as a regular X-Road message. For example, if there are 5 subscribers, 5 messages - one for each subscriber - are sent. The Message Room subsystem is used as the sending client, and the messages are signed with the publisher member's signing key. Each message is logged independently by the proxy. Steps 6-9 are repeated for each subscriber. If there are no subscribers, the message is removed without further processing.
The proxy sends the message to a subscriber.
The subscriber's Security Server processes the message and forwards it to the service (subscriber) defined in the request.
The service returns a confirmation that it has received the message.
The confirmation is returned as a regular X-Road response.
The Message Room Processor receives the response. The content of the response is ignored. However, if the response contains an error, the error is logged in the proxy log.

What’s next?

Based on the results provided by different approaches, the built-in approach has proven to be the most prominent. It can offer the same security guarantees that X-Road currently provides, including authentication, identity management, message logging, signing, and timestamping. Also, it can be expanded to a decentralized publish-subscribe implementation that enables many-to-many communication without a centralized message broker. Therefore, the next step is to test the PoC implementation of the built-in approach with a couple of selected use cases in Estonia and Finland during the first half of 2022. More information about the potential use cases is available in the previous blog post by Petri Kettunen.

Nevertheless, the NIIS members have not decided whether a production-level implementation of the Message Room concept will be included in X-Road. The PoC use cases implemented in Estonia and Finland will provide valuable information to support the decision-making. Once the PoC use case implementations have been completed, a decision regarding the production level implementation will be taken by the NIIS members. Meanwhile, NIIS participates in the implementation of the PoC use-cases and continues to develop the concept further.