Two Steps from X-Road REST Support

Over the last year I’ve written multiple blog posts about X-Road and REST. In those blog posts I have shared insights on our implementations plans, details about the technical design and status updates on the progress. In the last months we’ve concentrated on the technical implementation of the X-Road Message Protocol for REST and now we’ve arrived at a point when only the finishing touches are missing from completing the first stage of the REST support implementation. Therefore, in this blog post, I’m not going to write about plans, but the actual implementation instead. 

And for the readers interested in the technical details on source code level, the code implementing the REST support is now available in the develop branch of the X-Road master repository on GitHub. 

For clarity, adding support for REST does not mean dropping support for SOAP. No changes are required to information systems consuming and producing SOAP services via X-Road. Instead, the two architectural styles can co-exist side by side which means that all the current SOAP services are supported in the future too.

REST support implementation

For a recap, let’s start with defining what REST means in the context of X-Road. Unlike SOAP that is a protocol with a detailed specification, REST is an architectural style consisting of the best practices and guidelines. In X-Road’s case supporting REST means consuming and producing REST-style API’s via X-Road. A loose definition would be supporting any content type over HTTP.

One of the guiding principles in designing the X-Road Message Protocol for REST has been the ease of use for both service providers and service consumers. Therefore, consuming and producing REST-style services via X-Road is made possible without an additional adapter service component. X-Road-specific information required by Security Server (e.g. service client identifier, service provider identifier, message id etc.) is transferred and processed so that existing REST-style services and service consumers can be connected to X-Road with minimal changes or no changes at all. This has been achieved by transferring X-Road specific information required by Security Server in HTTP headers and URL parameters, outside of the message payload.

X-Road’s REST support is not limited to just JSON and XML messages as Security Server does not set any restrictions to the content type of the payload that is transferred between a service consumer and a service provider. The content type of the payload is defined using “Content-Type” HTTP header that is transferred between a service consumer and a service provider just like the payload itself. The payload is transferred as-is, Security Server does not modify, convert or validate the processed payload. The same goes with almost all the consumer and service provider defined HTTP headers and URL parameters – they are passed as-is between a service consumer and a service provider. The list of filtered HTTP headers is included in the X-Road Message Protocol for REST specification – all the other headers are passed as-is.

When it comes to non-repudiation of REST messages, message payload, URL parameters and HTTP headers are all included in the digital signature and logs generated and verified by Security Server. Hence, X-Road guarantees non-repudiation of REST request and response messages just like it does for SOAP messages. Currently it is possible to disable logging of SOAP message body and the same feature is available for REST services too. In that case REST message payload, URL parameters and consumer/provider defined HTTP headers are excluded from the message log.

Consuming REST services 

Consuming REST services via X-Road is simple – the service to be called is defined in the request URL and the X-Road client subsystem sending the request is defined in an HTTP header. Other X-Road specific information (e.g. user ID, issue, id) is optional and it is passed using HTTP headers. Other HTTP headers, path parameters and URL parameters are passed end-to-end as-is which means that from a service client’s perspective the only difference compared to a direct service call is the mandatory HTTP header defining the X-Road client subsystem.  

Providing REST services 

Producing REST services via X-Road is as simple (if not even simpler) as consuming services. Existing REST services can be published in X-Road as-is – it is enough to add the base URL of the REST API to be published and define access rights on the Security Server UI. Unlike with SOAP services, Security Server does not require X-Road specific information to be present in the responses returned by REST services. Certain X-Road-specific information is still included in the response message returned to a client information system, but Security Server takes care of adding the required information to response message’s HTTP headers. 

Service descriptions for REST 

Currently SOAP services must be described using WSDL descriptions. It is not possible to publish a SOAP service in X-Road without providing a WSDL description for the service.  

In the first X-Road version including REST support (v6.21.0) service descriptions for REST services are not required. When a new REST service is added, it is enough to provide the base URL of the REST API to be published. In later versions support for describing REST services using OpenAPI 3 specification will be added.  

How about automatic SOAP-REST conversions?

Services must be consumed using their native implementations – SOAP or REST. If a service provider wants to provide both SOAP and REST versions of the same service, the provider must implement both versions. In other words, Security Server will not provide automatic SOAP-REST conversion. In case automatic SOAP-REST conversion is needed, REST Adapter Service X-Road extension could be used. REST Adapter Service is an off-the-shelf component that provides an X-Road compatible REST-SOAP converter. The service supports a limited set of use cases. 

Machine-to-machine authentication 

The REST implementation supports mutual TLS authentication between a Security Server and a REST service consumer/provider. Support for JWT (JSON Web Token) based authentication between a Security Server and an information system may be provided in later versions. 

However, it is already possible to use JWT based authentication between a service client and a service provider. As described before, Security Server passes all HTTP headers between a service consumer and a service provider as-is, so there aren’t restrictions for implementing JWT based authentication on application level.

It’s time for beta! 

Soon it’s time to release the beta version of 6.21.0 that includes the long-awaited REST support. The official release version of 6.21.0 will be released at the end of April 2019. However, the beta version already provides all the REST-related features included in the final release. The last weeks are reserved for fine tuning and testing.

The version 6.21.0 will provide a basic support for consuming and producing REST services which includes:

• Basic REST functionality

  • Message exchange with signing and time-stamping

  • Message logging with archiving

  • Downloading and verification of log records

• Adding a REST service using an URL

  • No support for OpenAPI definitions

• Operational monitoring of REST services

• Service-level authorization

• Certificate based authentication (clients + services)

• X-Road Message Protocol for REST 1.0

That’s not all folks!

The REST support implementation will be done in phases which means that REST related features will be added along several X-Road versions – every new version adding something new. The next versions 6.22.0 and 6.23.0 will add more REST-related features later.

X-Road 6.22 (full support)

• Minor fixes and enhancements based on user feedback

• Metaservices for REST (listClients + listMethods)

• Support for OpenAPI

  • Add APIs using OpenAPI specification

  • Meta-service for querying services' OpenAPI definitions (getOpenAPIDefinition)

• Potential improvements

  • Path and method level authorization

  • JWT-based authentication (clients + services)

X-Road 6.23 or later (advanced support)

• Support for URI rewriting by Security Server

• Other API-Gatewayish features based on user feedback

X-Road Core Development in 2019

The new year 2019 has kicked off a few weeks ago, and X-Road core development keeps on rolling. Even if there haven’t been many updates regarding the X-Road core development lately, it doesn’t mean that NIIS has been resting on its laurels. Instead, we have been working on the version 6.20.0 and features for the other upcoming versions. We have also published the high-level X-Road development roadmap for 2019 so that everyone can see what kind of new features are coming out and when. The roadmap is available on the X-Road website.

The version 6.20.0 will be released on 25th January and it is the first of the three releases to be published in 2019. The official v6.20.0 release notes will be published in the X-Road Knowledge Base on the date of the release. The biggest change in the version 6.20.0 is support for Ubuntu 18.04 LTS operating system. The improvement is important, because the currently supported Ubuntu 14.04 LTS will quit receiving maintenance updates in Q2/2019 which is why migration is required. In addition, there’s a number of other improvements and fixes included in v6.20.0.

REST support

One of the most important features in 2019 is the long-awaited, native REST support. NIIS has been working on the X-Road REST support since spring 2018. Since then we have organized workshops and surveys for collecting input and feedback from X-Road users on the required features. We’re currently working on the implementation using all the comments and feedback as an input.

The implementation will be done in phases which means that REST related features will be added along several X-Road versions – every new version adding something new. The first version to support REST is v6.21.0, and it will be released in April 2019. Versions 6.22.0 and 6.23.0 will add more REST related features later.

X-Road 6.21 (basic support)

• Basic REST functionality

  • Message exchange with signing

  • Message logging

• Adding a REST service using an URL

  • OpenAPI definition not required

• Service-level authorization

• Certificate based authentication (clients + services)

• Metaservices for REST (listClients + listMethods)

• X-Road Message Protocol for REST 1.0

X-Road 6.22 (full support)

• Minor fixes and enhancements based on user feedback

• Path and method level authorization

• Support for OpenAPI

  • Possible to add a service using OpenAPI specification

  • Meta-service for querying services' OpenAPI definitions (getOpenAPIDefinition)

  • OpenAPI definition required

• JWT based authentication (clients + services)

X-Road 6.23 or later (advanced support)

• Support for URI rewriting by Security Server

• Other API-Gatewayish features based on user feedback

Right now, the version 0.3.0 of the Message Protocol for REST is out for comments – it is possible to leave comments until 28th January. All the feedback will be reviewed, and the protocol will be further developed based on the received input. NIIS welcomes everyone to comment the protocol draft!

API Based UI

Another major change taking place in 2019 is a new UI for both Security Server and Central Server. The new UI is not just a facelift – the UI will be redesigned and implemented from scratch. The aim of the change is to improve the usability and user experience of X-Road. Therefore, X-Road users will be closely involved in the design and implementation process so that their voice will be heard.

Together with the new UI, an administrative REST API will be implemented. The new UI will be using the API, but the API can be used for automating maintenance and configuration tasks too. The API will provide Security Server and Central Server administrators with the opportunity to automate maintenance and configuration tasks that currently require manual work.

According to the current schedule the new UI and the administrative API will be included in the version 6.22.0 that will be released in autumn 2019.

Streamlined onboarding process

Onboarding process of new X-Road members is one of the focus areas in X-Road core development this year. The aim is to streamline the onboarding process and reduce the number of steps that require manual work in the process. In practice this means enabling automatic approval of registration requests which speeds up the registration process and reduces the daily management tasks of the X-Road operator.

X-Road 7

The development of the core components of X-Road version 6 continues actively throughout the year 2019. At the same time NIIS is already looking ahead and the preparations for X-Road version 7 are already kicked off. Research and planning phase will be completed in 2019 so that the implementation can begin in 2020. Research and design are based on user-centered approach, and users are involved through the whole design and research process. Input, feedback and ideas are collected through interviews, workshops, events and surveys. In addition, one part of the research and planning phase is academic research in selected areas through collaboration with Estonian and Finnish universities.

And there is more!

This writing covers only the most important changes to be implemented in 2019. In addition, many smaller changes will be included in every release. It seems that we have a busy year ahead of us. If you’re interested in more detailed information about the upcoming changes, please visit the X-Road backlog. Anyone can access the backlog, and leave comments and submit enhancement requests through the X-Road Service Desk portal. Accessing the backlog and service desk requires creating an account which can be done in few seconds using the signup form.

Functional changes and new features implemented this year change the X-Road technology stack too. X-Road Tech Radar provides up-to-date information on different technologies used in X-Road.

Stay tuned!

Standalone Security Server

X-Road is a centrally managed distributed data exchange layer between Information Systems that provides a standardized and secure way to produce and consume services. The identity of each organization and Security Server is verified using certificates that are issued by a trusted Certification Authority (CA) when an organization joins an X-Road ecosystem. The identity of service producers and consumers is maintained centrally, but all the data is exchanged directly between a consumer and a provider.

X-Road’s distributed architecture makes it highly scalable and very resilient against different kinds of cyber attacks. X-Road creates a trusted network where message exchange takes always place between two trusted parties as the identities of all message exchange parties are verified using certificates. In general, these are major strengths of X-Road, but in some cases they’re also weaknesses, because they make the onboarding process of new members slower compared to a standalone solution that does not require any registration or verification to be completed. In production like environments certain controls are required as they enforce trust between member organizations. However, there are situations in which more agile approach is needed, e.g. testing and development purposes.

Currently the only way to be able to test Security Server is to join an existing X-Road ecosystem or set up an own environment. The time that is required for completing any of the two alternatives depends on the policies of an existing X-Road ecosystem and the experience level of the expert who is responsible for the task. Nevertheless, the time required for completing the task varies from hours to days. A developer who just wants to test a new service together with Security Server would like to have a solution that is available in minutes and requires minimal configuration. Sounds too good to be true - is something like that possible?

Is this a dream or is it now? 

Standalone Security Server is a special version of Security Server that is ready-to-use in minutes without the normal Security Server installation, configuration and registration process. It is meant for testing purposes in X-Road service development and it cannot communicate with other Security Servers. Therefore, it is targeted especially to developers and organizations that are developing services to be published via X-Road.

It is possible to add new services on the standalone Security Server and invoke the services using the same Security Server. It comes with two pre-configured subsystems – one for providing and another for consuming services. In addition, it does not require connection to Central Server, OCSP service or time-stamping service. Therefore, standalone Security Server can be set up in minutes and once it has been downloaded it does not even require an internet connection. It is ideal for testing purposes in service development or for someone who’s interested in giving a quick try.

Image 1. Standalone Security Server.

Standalone Security Server has not been published yet, but a proof of concept (PoC) level implementation has been completed by NIIS. Before publishing the standalone Security Server we would like to hear X-Road community’s opinion regarding the format in which it should be published. Do you want to have it as a Docker image, VirtualBox image, AWS AMI, Azure virtual machine image or something else? NIIS is going to publish a standalone version of Security Server using the format that receives the most votes.

NIIS welcomes everyone to give their vote by 1st November 2018!

X-Road REST Support – Where Are We Today?

As you already know, NIIS has been working on the X-Road REST support since spring 2018 when a survey regarding the topic was done. The results did not leave any room for doubts – 93 % of the participants wanted X-Road to support REST. After the survey NIIS organized an X-Road REST support ideation and planning workshop in Tallinn in May. The aim of the workshop was to get more insight on implementing REST support from real X-Road users. The feedback received from the workshop has been used as an input for the following phases of the planning process. Now the planning has reached a state where the designed implementation approach can be shared with the X-Road community. Let’s take a look where we are today.

From SOAP…

To be able to better understand the selected approach for implementing REST support, let’s have a look at the current SOAP based implementation first.

Image 1. Current SOAP based implementation.

Currently X-Road has two message protocols: X-Road Message Protocol and X-Road Message Transport Protocol. X-Road Message Protocol defines how service consumers and service producers communicate with Security Server. The protocol is based on SOAP profile 1.1 and it comes with some X-Road specific limitations and additional requirements, e.g. support for synchronous request-response operations only, some mandatory SOAP headers are required, document/literal style SOAP body is required.

Instead, X-Road Message Transport Protocol is a proprietary protocol that defines how Security Servers communicate with each other. The protocol uses HTTP 1.1 over TLS and MIME multipart framing. The protocol wraps the X-Road Message Protocol payload and adds some additional authentication data and message signature. The key limitation of the current implementation is that only SOAP payload is supported.

…to REST 

First, let’s define what REST means in X-Road’s context. Unlike SOAP that is a protocol with a detailed specification, REST is an architectural style consisting of best practices and guidelines. In X-Road’s case supporting REST means consuming and producing REST-style APIs via X-Road. A loose definition would be supporting JSON and/or XML over HTTP.

Image 2. Supporting SOAP and REST.

To be able to support REST-style services some changes to the current implementation are required. In addition to the current X-Road Message Protocol for SOAP, a new X-Road Message Protocol for REST will be created. The protocol will define how REST-style service consumers and service producers communicate with Security Server. The current X-Road Message Protocol for SOAP will remain unchanged – no changes are required to SOAP service consumers and service providers when REST support will be added. For clarity, adding support for REST-style services does not mean dropping support for SOAP services.

Adding support for REST-style services means that also X-Road Message Transport Protocol must be updated – a new extended version of the protocol will be created. The protocol will be changed so that the transport message will contain in place of SOAP request part a more generic payload part that can contain SOAP, JSON, XML etc. In fact, Security Server will not set any restrictions to the content type of the payload that is transferred between a service consumer and a service provider. The content type of the payload is defined using “Content-Type” HTTP header that is transferred between a service consumer and a service provider just like the payload itself.

Image 3. X-Road transport message before and after REST support is added.

In practice, the payload is transferred as-is – by default, Security Server does not modify, convert or validate the processed payload. However, validating that XML and JSON payloads are well-formed might be implemented. In general, services must be consumed using their native implementations – SOAP or REST. If a service provider wants to provide both SOAP and REST versions of the same service, the provider must implement both versions. In other words, Security Server will not provide automatic SOAP-REST conversion. In case automatic SOAP-REST conversion is needed, REST Adapter Service X-Road extension could be used. REST Adapter Service is an off-the-shelf component that provides an X-Road compatible REST-SOAP converter. The service supports a limited set of use cases.

The built-in REST support will have some limitations too. The implementation is restricted to request-response messaging model and it does not support HTTP streaming. Supporting these features would require some bigger changes to the X-Road architecture and therefore, they’re of out scope of REST support implementation.

Design goals 

Backwards compatibility is one of the most important design goals when implementing support for REST-style services. The behavior of SOAP payload will stay compatible with the current implementation. This means that no changes are required to information systems consuming and producing SOAP services via X-Road. 

When designing X-Road Message Protocol for REST, the aim is to design the interface for REST clients and services as future-proof. This means that the protocol will remain the same in the future versions of X-Roads which makes migrating to new X-Road versions significantly easier. Instead, the X-Road Message Transport Protocol, the protocol used between Security Servers, may change in the future versions of X-Road, but that does not have an effect on the service consumers and producers as long as X-Road Message Protocol for SOAP and REST remain unchanged.

One of the guiding principles in designing the X-Road Message Protocol for REST is the ease of use for both service producers and service consumers. In practice, this means that consuming and producing REST-style services via X-Road should be possible without an additional adapter service component. X-Road specific information required by Security Server (e.g. service client identifier, service provider identifier, message id etc.) should be transferred and processed so that existing REST-style services and clients can be connected to X-Road without making changes to services and clients to be connected. Therefore, X-Road specific information required by Security Server must be transferred outside of the message payload, in HTTP headers or URL parameters, for example.

Your input is needed!

The next phase in the REST support implementation process is a technical proof of concept (PoC) for validating the feasibility of the described approach before starting the actual implementation. The PoC has already started and the aim is to complete it in October.

Another task to be completed before starting the implementation is the definition of X-Road Message Protocol for REST. The protocol defines how REST-style service consumers and service producers communicate with Security Server. I have not covered the protocol in more detail in this blog post intentionally, because the first draft version of the protocol has just been completed and it is now available for comments. All the feedback will be reviewed, and the protocol will be further developed based on the received input.

NIIS welcomes everyone to comment the protocol draft! It is possible to leave comments until 18th October 2018.

X-Road Myth Busting – Part 1

Earlier this year I wrote a blog post about X-Road and blockchain which aim was to prove that there’s no blockchain in X-Road. The reason for writing the blog post was that many different sources were claiming X-Road to be a blockchain based technology – which is not true. The case around X-Road and blockchain should be closed now so it’s a good moment to bust some other myths and misunderstandings regarding X-Road.

X-Road origins

The X-Road was originally developed by the Estonian State Information Systems Department (at the Ministry of Economy and Communications) and the first version was launched in 2001. Many individuals from multiple organization were involved in the original implementation project. The creation of X-Road was a joint effort and multiple persons and organizations have contributed to it over the years. Therefore, calling a single person or an organization as the creator of X-Road would be an understatement for all the other contributors and participants. X-Road being created by a single individual or a single organization is clearly a myth.

Since the first version of X-Road the implementation technologies and software versions have changed as the years have gone by. X-Road is not based on any other data exchange solution or software – the current version 6 is based on Java 8, Ruby, Akka and several other technologies. X-Road is released under the MIT licence and is available free of charge for any individual or organization. The MIT licence is a permissive free software licence that puts only very limited restriction on reuse – also reuse in closed commercial products is permitted. However, X-Road is a registered trademark of the Estonian Information System Authority (RIA) and the trademark can only be used under a licence granted by RIA. Therefore, any derivative works and services based on X-Road software cannot use the name X-Road without a proper licence granted by RIA.

There are commercial data exchange layer products that are based on X-Road and provide X-Road compatible protocols. Some of these products are claiming to be the core technology of X-Road even if their first versions have been released after the first version of X-Road saw daylight. Based on this, X-Road using another data exchange solution or software as its core technology is a myth.

Distributed ledgers and databases

Some sources have claimed X-Road to be a distributed ledger or a distributed database. The key idea behind both technologies is that they are used for storing data in a distributed way – the data is shared across multiple nodes and all the nodes have their own identical copy of the data. All the changes in the data are replicated to all the nodes in the network. Technical implementation details and data replication mechanisms vary between distributed ledgers and distributed databases, so they should not be understood as synonyms. Underlying distributed ledger is blockchain, whereas distributed databases are based on different kind of database management systems.

X-Road is a centrally managed distributed data exchange layer between information systems. Identities of message exchange parties are maintained centrally, but all the data is exchanged directly between a consumer and provider. Central Server contains a registry of X-Road members and their Security Servers. Central Server can be clustered and in that case the contents of the registry are replicated across all the Central Server nodes using database replication techniques. The registry is based on a relational database and there’s no blockchain in its implementation.

The key idea of X-Road is to support decentralized data management. It means that each service provider owns its data and instead of maintaining multiple copies of the data across different information systems the data should be requested from the owning data source directly. The data is always transferred between a service consumer and service provider, and it is not stored centrally or replicated between Security Servers.

The common factor between distributed ledgers, distributed databases and X-Road is distributed architecture. Despite the same architectural model and some shared design goals like high availability and security, distributed architecture is used for implementing very different features and functionalities. Distributed ledgers and distributed databases are used for storing data and they replicate it across multiple nodes, whereas X-Road itself does not store data – it provides a secure data exchange channel between the owner of the data and multiple consumers. X-Road uses distributed database technologies within its clustered components for high availability, but the replication covers configuration data only. For these reasons, X-Road being a distributed ledger or a distributed database is a myth too.

End-user authentication

As already described before, X-Road is a data exchange layer between information systems. Among other things, X-Road provides organization level and machine level authentication that is based on Public Key Infrastructure (PKI). The identity of each organization and Security Server is verified using certificates that are issued by a trusted Certification Authority (CA) when an organization joins an X-Road ecosystem.

In case X-Road is used as a data exchange layer in a process that involves end-users and require their authentication, service consumer and service provider are responsible for the authentication of the end-user. Usually, the service consumer must authenticate the user before sending a request via X-Road and then it’s up to the service provider to decide whether it requires some evidence regarding the authentication to be sent as a part of the service request, e.g. authentication token, session context etc. From X-Road’s point of view end-user authentication is completely transparent and in case some data regarding the authentication is sent within the messages X-Road does not verify or validate it in any way.

As also in the previous cases, X-Road implementing end-user authentication is a myth.

There is more?

In this blog post I have covered some most common myths and misunderstandings that we at NIIS face on a regular basis. There are probably more of them and new ones are also born now and then. Therefore, this post is titled as X-Road Myth Busting – Part 1. The series of X-Road myth busting will probably continue in the future.

X-Road as a Platform to Exchange MyData

The current way of how personal data is managed is often called organization centric as organizations are managing the data and controlling how and by whom it can be used and accessed. MyData has a different approach as it aims to move towards a human centric system that allows an individual to access and control his or her personal data. Personal data is considered to be MyData if it is under the respective individual’s own control. Therefore, not all personal data is MyData, but all MyData is personal data. According to MyData principles individuals should have the right and practical means to manage their personal data, the data should be technically easy to use and accessible, and there should be a shared infrastructure in-place that enables transfer and management of the data.

Technically, the aim of MyData is not to build one centralized data storage for personal data, but rather create a distributed model where data is stored in different information systems and registers that communicate with each other through APIs (Application Programming Interfaces) using a standardized messaging model and shared, open infrastructure. The data should flow directly between a data source and a data using service. In addition, datasets should be interoperable and portable between information systems and sectors. Access to datasets should be controlled through consents by an individual.

Image 1. MyData model and roles.

In addition to the individual, the data subject, the model has other three key roles: MyData operator, data source and data using service. MyData operator maintains MyData accounts that store consents created by individuals. A consent gives a data using service an authorization to access an individual’s personal data provided by a data source. In other words, an individual uses the account for granting (and revoking) services access to his or her personal data. Through the account an individual should also be able to see the access logs regarding his or her personal data — which data using service has accessed what information, when. A data source is an information system that stores personal data and provides access to it through APIs. Instead, a data using service is an information system that consumes personal data provided by one or more data sources. Both data sources and data using services must implement the required MyData APIs and enable their services to be connected with MyData accounts.

In the big picture, there will be multiple MyData operators and an individual may have multiple MyData accounts. However, the aim of the MyData architecture is to make the accounts interoperable which means that individuals should be able to change their MyData operator service. Achieving interoperability requires design and standardization on e.g. data formats, trust networks and semantics. In practice, this means that MyData operator services must use the same data models and semantics, implement the same APIs etc.

X-Road Data Exchange Layer

MyData requires a secure, standardized and mature platform technology — that’s where X-Road comes into play. X-Road is an open source data exchange layer solution that enables organizations to exchange information over the Internet. X-Road is a centrally managed distributed data exchange layer between information systems that provides a standardized and secure way to produce and consume services. X-Road ensures confidentiality, integrity and interoperability between data exchange parties.

Image 2. X-Road data exchange layer roles and components.

X-Road is used nationwide in the Estonian public administration and in the Suomi.fi Data Exchange Layer service. X-Road is released under the MIT licence and is available free of charge for any individual or organization. Nordic Institute for Interoperability Solutions (NIIS) is responsible for the development of the X-Road core and providing support and insights to all the interested parties.

X-Road implements a set of common features to support and facilitate data exchange. X-Road provides the following features out of the box:

• address management

• message routing

• access rights management

• organization level authentication

• machine level authentication

• transportation layer encryption

• time-stamping

• digital signature of messages

• logging

• error handling.

The identity of each organization and technical entry point (Security Server) is verified using certificates that are issued by a trusted Certification Authority (CA) when an organization joins an X-Road ecosystem. The identities are maintained centrally, but all the data is exchanged directly between a consumer and provider. Message routing is based on organization and service level identifiers that are mapped to physical network locations of the services by X-Road. All the evidence regarding the data exchange is stored locally by the data exchange parties, and no third parties have access to the data. Time-stamping and digital signature together guarantee non-repudiation of the data sent via X-Road.

Image 3. X-Road messaging model.

An X-Road ecosystem is a community of organizations using the same instance of the X-Road software for producing and consuming services. The owner of the ecosystem, the governing authority, controls who’s allowed to join the community, and the owner defines regulations and practices that the ecosystem must follow. The ecosystem may be nationwide, like in Estonia and Finland, or it may be limited to organizations matching certain criteria, e.g. clients of a commercial service provider. Technically, the X-Road software does not set any limitations to the size of the ecosystem or to the member organizations.

Two X-Road ecosystems can be joined together, federated. Federation is a one to one relationship between two ecosystems. Members of the federated ecosystems can publish and consume services with each other as if they were members of the same ecosystem. It is possible to create federation connections with multiple ecosystems, but transitive federation relationships are not supported. Ecosystem does not have a federation relationship with another ecosystem that it’s not directly federated with.

MyData via X-Road

After describing the requirements for MyData platform technology and the key features of X-Road data exchange layer solution it seems that X-Road might be a good match to the requirements. However, until now we’ve been talking about MyData and X-Road on a high level, but to be able to evaluate the match between MyData and X-Road better, we must dive into the details.

Image 4. MyData via X-Road.

X-Road can be used as a data exchange layer between data providing services, data using services and MyData operator — for transferring both consents and data. In addition, access logs created by X-Road are transferred from X-Road Security Servers to MyData operator via X-Road too. In fact, all data exchange between different data exchange parties is implemented using X-Road’s standard message exchange protocol. In this way, X-Road takes automatically care of several security controls that otherwise must be implemented separately for every new connection between data source and data using service, data source and MyData operator, and data using service and MyData operator.

The use of X-Road requires that all the message exchange parties are members of an X-Road ecosystem and they have access to an X-Road entry point, Security Server, which is required for both producing and consuming services. The Security Server mediates service calls and service responses between information systems, and it encapsulates the security aspects of the X-Road infrastructure: managing keys for signing and authentication, sending messages over secure channel, creating the proof value for messages with digital signatures, time-stamping and logging.

X-Road logs contain all the messages processed by the Security Server. Each message is time-stamped and signed which makes it possible to verify the message content afterwards. By default, the logs are stored locally by the Security Server — external parties do not have access to them.

Both data using services and data sources are identified using X-Road’s globally unique identifiers. The identifiers contain information about the X-Road ecosystem, member organization and information system that is consuming or producing data via X-Road. The identifiers of data using services consist of 4 parts (1–4) and the identifiers of data sources consist of 6 parts (1–6):

1. X-Road instance identifier — identifier of an X-Road ecosystem.

2. Member class — type of member organization, e.g. governmental (GOV), commercial (COM).

3. Member code — business ID of a member organization.

4. Subsystem code — identifier of an information system connected to X-Road.

5. Service code — service code of a service that can be consumed via X-Road.

6. Service version — version number of the service.

The identifiers are used internally by X-Road for routing messages between data using services and data sources. A data using service does not need to know the network address of a data source — X-Road automatically maps the service identifier to the correct network address.

Built-in features provided by X-Road can also be used for managing access rights to data sources. Access rights management is based on the X-Road service identifiers. The key idea of X-Road is that each service provider owns its data and is responsible for managing access rights of its services. In other words, publishing a service to X-Road does not mean that the service is automatically accessible to all X-Road member organizations. Usually, access rights are granted on information system level — a service provider grants a specific information system access to a service. This approach provides a tight control, but it may also generate lot of administrative work if the number of service consumers is changing frequently.

Image 5. MyData access rights management based on X-Road service identifiers and global groups.

X-Road provides a feature called global groups that allows a service provider to grant a specific group of information systems access to a service. All the information systems that are members of a global group will then have access to the service. The members of the group are centrally managed by the X-Road governing authority so it’s possible to limit the membership of a group to organizations meeting certain criteria. This allows creating a global group for MyData data using services that contains only information systems that are registered as a data using service by a MyData operator. MyData data sources can then grant the global group access to their MyData services. In this way, all MyData data using services have access to all the MyData data sources through a single group membership. To make this work in practice, a process for synchronizing MyData data using services between the MyData operator and the X-Road governing authority is required. Technically, global groups are maintained on the Central Server operated by the governing authority and automatically distributed to all the Security Servers.

Image 6. MyData request flow via X-Road.

Using global groups data using services can be granted access to all MyData data sources. However, data using services are not allowed to fetch just any individual’s data — they need to be authorized by the individual to do so. Before requesting an individual’s data from a data source, a data using service must check that a consent created by the individual exists in MyData operator’s database. Without the consent a data using service has no authorization to request an individual’s data from a data source. The consent may be checked by a data using service only, or by both a data using service and a data source. If the consent is checked by a data using service only, a data source trusts that the data using service is authorized to access the individual’s data. From X-Road’s point of view both approaches are supported, and they can even co-exist within the same X-Road ecosystem. Either way, unauthorized use of an individual’s data can be automatically detected by analyzing the logs that are collected by the MyData operator and comparing them to existing consents. Unauthorized use of data may be subject to penalties, e.g. exclusion from the service etc.

The Perfect Match?

MyData requires a secure, standardized and mature platform technology — and X-Road meets the requirements. X-Road creates a trust network where the identity of each organization and access point is known and can be verified. In addition, X-Road provides several built-in security features that make data exchange secure and traceable and guarantee non-repudiation of the data sent via X-Road. Access rights management between data using services and data sources can also be implemented using an X-Road provided feature, global groups. In addition, X-Road supports the model of multiple MyData operators and MyData accounts.

X-Road is based on distributed model which makes it resilient against cyberattacks and service interruptions. The first version of X-Road was released in Estonia in 2001 and the concept has proven to be successful even if the implementation technologies and software versions have changed over the years. In addition to Estonia, X-Road has been implemented in Finland and many other countries all over the world. Different X-Road ecosystems can be joined together, federated, which makes it possible to consume and provide services between ecosystems. This feature can be used to exchange MyData between X-Road ecosystems too, e.g. cross-border MyData exchange between Estonia and Finland.

X-Road is an open source technology, and anyone has access to it for free of charge. An X-Road ecosystem is managed by a governing authority that controls who are allowed to join the ecosystem. In Estonia and Finland the ecosystems are open for all kind of organizations (public, private, non-profit etc.) and joining them is free. Joining an X-Road ecosystem does not require an organization specific Security Server as a Security Server can be shared between multiple organizations or provided as a service by a third party, e.g. a commercial service provider. The process that is required to join an ecosystem is more time consuming compared to implementing a direct point-to-point integration, but it is necessary for creating trust between the members of the ecosystem. In addition, the process is required only when a new organization joins the ecosystem or a new Security Server is registered.

Despite everything X-Road can offer, it alone is not enough to answer all the open questions regarding MyData platform technology and interoperability. X-Road provides a secure and standardized way to exchange MyData, but it does not enforce semantic interoperability, common business data models and standard business APIs. X-Road provides a framework for data exchange, but it does not process or transform the business data that is transferred — that is out of its scope. In addition, X-Road provides a technical framework for service level access rights managements, but to make the model work in practice, a legal framework for service agreements between data sources and data using services is required too. Besides that, X-Road does not provide implementation of MyData operator and consent verification either, so they must be implemented separately. X-Road also sets some requirements to their implementation as the service identifiers used by X-Road must be included in the MyData operator’s data model and consents. In addition, also X-Road requires some further development to make implementing all the presented features possible. For example, X-Road does not currently provide a mechanism for sharing log entries in a way that enables accessing them from an external system and providing a centralized view to access logs.

All in all, X-Road provides many features that are required from MyData platform technology. Instead of developing everything from scratch, X-Road provides an alternative with a solid starting point that enables targeting resources to missing components and open questions. X-Road is an existing and operational system which allows testing new ideas and approaches from day one instead of waiting for weeks or months.

NIIS is responsible for developing X-Road core technology, and welcomes everyone to submit new ideas and enhancement requests regarding X-Road and MyData. X-Road’s backlog is public — anyone can access it, leave comments and submit enhancement requests through the X-Road Service Desk portal. Accessing the backlog and service desk requires creating an account which can be done in few seconds using the signup form.

This article was originally published in the MyData Journal on the 31st August 2018.

NIIS Begins X-Road Core Software Development

The X-Road was originally developed by the Estonian State Information Systems Department (at the Ministry of Economy and Communications) and the first version was launched in 2001. Estonia and Finland developed the X-Road core together from 2015 until June 2018 when the development was handed over to NIIS. The handover caused changes in the locations of source code repositories and the joint development model was updated as well.

Since June 2018 NIIS has been managing the X-Road core technology and finalizing the preparations regarding the beginning of the actual development activities. The development of the X-Road core was kicked off by NIIS on 7th August when sprint 1 officially started! Now it’s a good moment to take a step back and have a look what was required to get to this point.

Building the capabilities

Building the capabilities required for taking over the ownership of the X-Road core was started in February 2018. The task may sound trivial, but it’s good to remember that we’re talking about an information system that is the backbone of nationwide e-government infrastructure in multiple countries. Therefore, there was no room for error and the handover of the X-Road core had to be completed seamlessly.

First of all, NIIS needed technical capabilities for agile software development and expertise to do the actual development work too. NIIS’s strategy was to use outsourcing, and procurement processes for acquiring these capabilities were started in March 2018. Technical capabilities included development tools and environments for agile software development. As X-Road core had been developed for years by Estonia and Finland, the tools to be used were chosen based on the previously used tools and platforms to enable reuse of already existing utilities and resources.

Development tools and process

Once the service provider for the development tools was selected through a procurement process the actual work for setting up and configuring the required tools was started. The tools were handed over to NIIS in May 2018 and the project for transferring X-Road core development and testing environments to NIIS was started immediately. However, some tasks needed to wait until NIIS had its own development team available. At the same time the procurement for the X-Road core software development resources was proceeding as planned and the original goal to be able to start the X-Road core development in summer 2018 was coming closer and closer all the time.

Image 1. NIIS development tools.

During the previous years, Estonia and Finland had developed a well defined process for running the joint development. NIIS needed to be added to the picture which meant re-thinking the process and roles and responsibilities of different parties. A bilateral process had to be changed to multilateral so that the process wouldn’t have to be changed again in case new parties were to join in at some later point. The process was updated during spring 2018 and the new updated version was published at the same time when the X-Road core source code repositories were handed over to NIIS in June 2018.

Completing the preparations

The X-Road core software development procurement was finalized in June 2018 and Gofore Plc was selected as a cooperation partner to develop the X-Road core. Now NIIS had its own development team and development tools, but the tools still required some configuration and the transfer of the development environments needed to be finalized. The environments are hosted on Amazon Web Services (AWS) cloud platform and the process for setting them up is highly automated. However, also automation scripts and templates need to be cleaned up and refactored now and then, so the transfer did not mean just moving all the existing environments as-is, but updating, optimizing and refactoring the environments too.

Image 2. X-Road development, testing and distribution

Recent years of X-Road development have shown the importance of tooling and automation in the development and testing process which is why it was decided that sprint 0 (2nd July - 6th August 2018) was dedicated for the setup of tools and environments. In this way, the development of new features could be started from sprint 1.

The work ahead

During the first half of 2018 NIIS built required capabilities for taking over the X-Road core software development. The first part of the year was extremely busy and most likely the second part will be even busier as the X-Road core development has now started.

The most important backlog items for the second half of the year are support for Ubuntu 18.04 LTS operating system and native support for consuming and producing REST services. In addition, smaller backlog items will be implemented in every sprint and the backlog will be constantly prioritized by the Working Group.

Anyone can access the backlog and development board, leave comments and submit enhancement requests through the X-Road Service Desk portal. Accessing the backlog, development board and service desk requires creating an account which can be done in few seconds using the signup form.

X-Road Core Software Testing

Testing plays an important role in every software project and its aim is to verify that the software meets its requirements and find errors and other defects in the software. X-Road is no exception – the data exchange layer is a critical component of e-government architecture and it is used nationwide in multiple countries so there’s no room for software errors.

In a traditional waterfall software development project, testing is typically done during a separate testing phase once requirements have been defined and the code implementing them has been programmed. In agile software development projects testing is usually done concurrently with requirements and programming. The benefit of the agile approach is that problems are detected faster which makes fixing them easier and therefore also cheaper. X-Road core is nowadays developed using agile development methods and testing follows the same approach. The aim of X-Road core testing is to automate everything that can be automated and catch all the errors as early as possible. Automation helps to reduce manual work, but it also ensures that processes are repeated in the exact same way each time.

Background

X-Road has around 50 000 lines of code so manual testing is really not an option. The Finnish Population Register Centre and Estonian Information System Authority, that jointly developed X-Road before the development was handed over to NIIS in June 2018, invested a lot of effort in automated testing and code quality in the recent years. For example, unit test coverage increased from 29 % (2015) to 40 % (2018). At the same time technical debt ratio decreased from 5.3 % to 2.6 %. Total test coverage also increased significantly, but unfortunately no exact numbers are available as measuring it is still a work in progress.

X-Road core testing now

The tests were handed over to NIIS together with X-Road core in June 2018. NIIS is responsible for testing the X-Road core that includes Central Server, Configuration Proxy, Security Server and external APIs and interfaces provided by the core components. Organizations operating their X-Road environments are responsible for testing any specific hardware and their own extensions built on top of the X-Road, e.g.: API catalog, environmental monitoring tools, operational monitoring tools etc.

Image 1. X-Road development and testing tools

Just like the software requirements, test cases and test results are managed in JIRA issue management system. Most of the tests are automated and there are three alternative execution schedules:

• when new code is submitted

• daily

• before releasing a new version.

The execution schedule of a test depends on the type and nature of the test. For example, unit tests are executed every time when new code is submitted, but performance tests are executed daily. There is no sense to execute all the tests on every change only because it can be done.

Automated tests are divided in the following categories:

• unit testing

• API / interface testing

• UI testing

• performance testing

• security testing

• source code quality control and static analysis

• installation and upgrade testing.

New releases must pass all the categories and tests before they are published and distributed to NIIS members and X-Road community.

Image 2. X-Road development, testing and distribution

In addition to automated testing, some areas require manual work:

• internal code reviews – developers review each other’s work (ongoing)

• security audits – source code + white hat hacking (periodical)

• licence compliance checks (periodical).

X-Road core development and testing is done on Amazon cloud which enables optimizing resource usage and maximizing automation. In practice, this means that all the required resources can be created and destroyed automatically, and resources need to exist only when they are actually being used. For example, there’s no need to clean up a performance test environment between test runs as a brand new environment can be automatically created for each run.

The road ahead

X-Road is based on distributed architecture and many use cases contain complex processes that require operating multiple components asynchronously, e.g. registration of a new Security Server requires operating Central Server, Security Server and Certification Authority. Despite the complexity most of the use cases have automated tests that can be executed without manual work as often as it's required. When the software changes the tests must be changed accordingly, of course. Just like software development, also testing is continuous work that is not about to end any time soon.

NIIS will continue to develop all the aspects of the quality assurance of X-Road. Currently NIIS is looking for a testing partner to provide resources for X-Road core software testing. The procurement is open until the end of July 2018 and general information about the procurement can be found in the Estonian e-procurement portal at https://riigihanked.riik.ee/register/hange/197345.

Balancing the Load in X-Road

In general, load balancing means distributing workloads across multiple computing resources. Instead of relying a single resource the same service is deployed on multiple resources and service requests are distributed across all of them. If one of the resources stops responding, no more requests are routed to it and other available resources take care of serving the requests. Load balancing is used to increase performance and availability using multiple components instead of a single one. Load balancing can be implemented in different ways – a load balancer can be software or hardware based, DNS based or a combination of the previous alternatives. In addition, load balancing can be implemented on client or server side.

X-Road Security Server has an internal client-side load balancer and it also supports external load balancing. The client-side load balancer is a built-in feature and it provides high availability. Support for external load balancing has been available since version 6.16.0 and it provides both high availability and scalability from performance point of view.

Internal load balancing in X-Road

The internal client-side load balancer is a built-in feature of the X-Road Security Server and it is operating on “Fastest Wins” basis. When a service is registered on multiple Security Servers (same organization, same subsystem, same service code) the server that responds the fastest to TCP connection establishment request is used by a client Security Server. Once a provider Security Server is selected, it will be used for subsequent requests by the client until the TLS session cache expires or a connection attempt fails.

Image 1. “The Fastest Wins” - the server that responds the fastest to TCP connection establishment request is used by a client Security Server.

If the fastest Security Server providing the service quits answering, the client Security Server will automatically change to the second fastest and so on. Connections are evaluated on TCP level only so higher-level application related problems are not taken into account.

Image 2. If the fastest service provider fails, the client will automatically change to the second fastest.

The solution provides high availability, but not scalability from performance point of view as the load is not evenly distributed between all the provider side Security Servers. However, this does not mean that all the different client Security Servers would use the same provider Security Server. The client Security Server prefers the provider Security Server that is nearest network-wise (round trip time is lowest) and the fastest provider varies between different clients. In other words, load generated by different client Security Servers is distributed between different provider Security Servers, but the distribution is not based on a load balancing algorithm so there’s no guarantee that the load is distributed evenly.

Image 3. Load generated by different client Security Servers is distributed between different provider Security Servers.

When relying on internal load balancing adding a new node means installing and registering a new Security Server. Each Security Server serving the same service has its own identity which means that it has its own authentication and sign certificates. In addition, each Security Server providing the same service is stand-alone and there’s no automatic synchronization regarding registered services and/or service level access permissions between Security Servers. Maintaining and synchronizing configuration between Security Servers is a manual task.

Internal load balancing is completely transparent to the client-side information system as the client Security Server takes care of routing the requests, verification of certificates etc. internally. For the client-side information system it’s enough to send a request to the client-side Security Server and it will take care of the rest using the global configuration data provided by the Central Server for discovering Security Servers that provide the requested service. This makes it possible to add new provider Security Servers and/or change the network location of existing provider Security Servers without making any changes on the client-side. High-security environments, where all the outgoing network connections are blocked by default and only connections to whitelisted targets are allowed, are an exception as all the new provider side Security Servers must be explicitly whitelisted on firewall configuration.

External load balancing in X-Road

First, let’s define the meaning of external load balancing. In this context, external load balancing means that a third-party software or hardware-based load balancer (LB) is used for distributing load (LB 1-2) between an information system and the X-Road Security Server or (LB 3) between Security Servers. There are three different use-cases that include an external load balancer:

• A load balancer (LB 1) between service consumer(s) and a Security Server cluster.

• A load balancer (LB 2) between service providers and a Security Server cluster or a single Security Server.

• An external internet-facing load balancer (LB 3) that distributes inbound requests from other Security Servers to a Security Server cluster.

Image 4. An external load balancer can be used in three different scenarios. Different scenarios can also be combined.

The first two scenarios are about distributing load between the Security Server and an information system and they have always been supported by the Security Server. Therefore, we’re not going to concentrate on them now. Instead, let’s take a better look at the third use-case.

The Security Server has supported the use of an external internet-facing LB since version 6.16.0. In this setup an external LB is used in front of a Security Server cluster and the LB is responsible for routing incoming messages to different nodes of the cluster based on the configured load balancing algorithm. Using the health check API of the Security Server the LB detects if one of the nodes becomes unresponsive and quits routing messages to it. The solution provides high availability and scalability from performance point of view.

Image 5. An external LB can be used in front of a Security Server cluster and the LB is responsible for routing incoming messages to different nodes.

A Security Server cluster can have undefined number of nodes which are all active (not hot standby). From client Security Server’s point of view a cluster looks like a single Security Server as all the nodes have the same identity (server code, certificates etc.) and they’re all accessed using the same public IP that is registered to the global configuration as the Security Server’s address. Therefore, a cluster is completely transparent to client-side information systems and Security Servers.

When a clustered Security Server acts as a client and makes a request to an external server (Security Server, OCSP, TSA, Central Server), the external server sees the public IP address. However, the public IP address used for outgoing requests might be different from the one used for incoming requests.

Image 6. When a clustered Security Server acts as a client and makes a request to an external server, the external server sees the public IP address.

One of the nodes is the master node and all the other nodes are slaves. Maintaining a cluster’s configuration is easy, because configuration changes are done on the master and they’re automatically replicated to the slaves. Replication covers the configuration database and configuration files. Changing the configuration on the slaves is blocked. However, all nodes fetch global configuration and OCSP responses independently. In addition, message log database is Security Server specific and it is not replicated between nodes.

Image 7. Serverconf database and configuration files are automatically replicated from the master to the slaves.

Support for an external LB has not been designed some specific LB solution in mind. Any software or hardware based LB that supports HTTP health check and load balancing TCP traffic can be used, e.g. AWS ELB, F5, HAProxy, Nginx. The LB uses the health check API of the Security Server for checking the state of all the nodes in a cluster. The health check API has been available since version 6.16.0 and it can be used for monitoring purposes as well. Health check returns HTTP 200 OK when security server is operating normally, otherwise HTTP 500. The health check API must be enabled manually as it is disabled by default.

The use of an external internet-facing LB also enables dynamic scaling of a Security Server cluster. Dynamic scaling means that the number of nodes in the cluster can be automatically adjusted based on the selected metrics, e.g. CPU load, throughput of incoming requests etc.  Scaling can also be done based on a predefined schedule – the number of nodes varies between time of the day and day of the week, e.g. number of nodes is increased during peak hours and decreased for the night. As all the nodes in the cluster share the same identity, adding a new node can be automated because it doesn’t have to go through the normal registration process that requires manual work. Changing the number of nodes can be done creating/deleting nodes or starting up/shutting down existing nodes. Either way, less resources are consumed compared to a situation where enough resources for handling the peak load are running 24/7.

Clustering enables dynamic scaling of Security Servers, but the X-Road does not provide off-the-shelf tools for the implementation as the implementation is platform specific. For example, Amazon Cloud Services (AWS) Auto Scaling and Elastic Load Balancer (ELB) services can be used to implement dynamic scaling and other cloud platform providers (Microsoft Azure, Google Cloud Platform etc.) have similar services.

Which one to use?

Compared to the X-Road's Security Server’s internal load balancing feature, an external load balancer provides better support for high availability and scalability from performance point of view. External load balancer gives the provider side Security Server owner full control of how load is distributed within the cluster whereas relying on the internal load balancing leaves the control on the client-side Security Servers. However, setting up a Security Server cluster is more complicated compared to internal load balancing that is a built-in feature and enabled by default. In addition, an external load balancer brings additional complexity to Security Server version upgrades too as the upgrade process must be coordinated within the cluster. On the other hand, adding new nodes to a cluster is easy as the normal registration process is not required, because all the nodes in the cluster share the same identity. When relying on the internal load balancing each node is independent and has its own identity – adding a new Security Server means that full registration process must be completed.

In addition, external load balancing has one great benefit compared to internal load balancing. The Security Server health check API used by an external LB recognizes situations where a Security Server is running, but PIN code is missing. When PIN code is missing, the Security Server is not able to process messages. Internal load balancing is not able to recognize this situation as it is operating on the TCP level – establishing a TCP connection between Security Servers works even if the PIN code is missing and the provider server is not able to process messages. Therefore, internal load balancing might route messages to Security Servers that are not able to process them because of a missing PIN code. This kind of situation might happen after a Security Server has been restarted, but the administrator hasn’t entered the PIN code yet. Fortunately, there’s an easy solution to the missing PIN code problem. Entering PIN code can be automated using xroad-autologin add-on that can be used together with both built-in and external load balancing.

After discussing different alternatives and combinations the next question probably is when different solutions should be used. The bad news is that there is no single right way to do it. The best solution is always case specific and it varies between different use-cases and information systems. Requirements regarding the availability, scalability and performance of the information system must always be taken into consideration and the solution should be designed based on them. Different alternatives can be and should be used together to provide the best overall solution.

X-Road Logs Explained – Part 3

This is the third post in a series about the X-Road logs. The first part was about different log types (technical logs, business logs, audit logs) and the X-Road logs in general. The second part concentrated on the X-Road business log which contains all the messages processed by a Security Server. The third part is about how to provide access to the logs of who has accessed my data and when.

Our data out there

Our personal information is processed by numerous different information systems on a daily basis. Some of the processes are fully automated and other include human actions. Wouldn’t it be nice to know for which purposes your personal information is used, when and by whom? Getting access to this information is already possible, at least in theory, but it requires requesting the information from each registry owner separately. Once you get a response, its format (printed document, email, structured data etc.), level of detail and delivery method (post, email, API etc.) depend on the registry owner, because there’s no unified way of providing this information. In practice, it is currently impossible for a citizen to get a good overall picture regarding the usage of his/hers personal information. There must be a better way.

As you all know by now, the X-Road message log contains all the business events processed by a Security Server. Non-repudiation of the message log is guaranteed using time-stamping and digital signatures. Could the message log be used for providing access to the information regarding who has accessed my data and when? Let’s find out.

X-Road message log to rescue

The message log contains all the required information in a machine-readable format so it might provide a solution to our problem. However, message log is Security Server specific so when a service can be accessed through multiple Security Servers, which is very common in a highly available setup, message log entries are distributed between multiple Security Servers. This means that information must first be collected from multiple sources and then combined. Collecting and combining the information requires shell access to Security Server as there’s no API for querying the logs based on the message content. In practice, collecting and combining the information must be done manually by a Security Server administrator.

Another thing that must be considered is the archiving of the Security Server message log. By default, messages are archived from message log database to disk after 30 days and therefore there must be an additional method for searching data from the archived message log files. This can be done using standard Linux command line tools, but it’s not very efficient when the amount of archived log data is big. In addition, it’s not recommended to keep archived logs on the Security Server so access to a separate long term storage is required and the archived logs might even be encrypted.

Too good to be true?

It seems that all the required information is there in the message log and with some new functionalities the information could be made accessible in a way that manual work is no longer required. Sounds like a good solution if personal information which usage needs to be logged is accessed through the X-Road only. In real life, this is rarely the case. Usually personal information is accessed through multiple channels (information system’s own UI, mobile apps, p2p integrations using native APIs etc.) and the X-Road is just one of them. The message log contains information about the messages processed by a Security Server, but all the other channels are excluded. From a citizen’s point of view this kind of partial solution is not sufficient and usage logs must contain the access information from all the different channels. To be able to log all the events processed through different channels, an alternative approach is needed.

The technical challenges mentioned above could be solved using a centralized system for storing usage logs. In general, it is a common practice for an organization to have a centralized log management system that contains logs from organization’s all the information systems. A log management system could be designed so that it provides a separate index/table/container for information related to access to personal information. In addition, a common access log format, that all information systems would use, should be defined and implemented. However, unlike Security Server’s message log, information systems’ logs are not usually signed and time-stamped so non-repudiation is not guaranteed.

Accessing usage logs

Implementing a centralized logging system on organizational level would solve half of the problem - collecting and storing the access logs. The other half, providing citizens access to the usage logs is yet to be resolved. Each organization could build their own solution for viewing the data, but for a citizen getting an overall picture regarding different services would require accessing many different online services and websites. A better solution would be to provide access to all the usage logs from one centralized service or portal, e.g. state portal.

Image 1. Centralized access to usage logs.

Centralized access to all usage logs could be implemented in a distributed or centralized way. The distributed way means that each organization stores its own usage logs and the data is fetched on request only, e.g. when a citizen logs into the state portal and wants to see the access logs of a specific data source. The centralized way means that usage logs are regularly harvested from organization specific storages and stored in one central usage log storage. Both alternatives have their pros and cons, but their further analysis is out of scope of this blog post. For a citizen both alternatives would provide the same result – access to all the usage logs from one place.

Access to the logs through state portal could be implemented using the X-Road. Each organization would implement a common interface for accessing and/or harvesting logs. In this way, even the access to the access log would be logged and transparent to citizens.

Of course, in addition to architectural and technical questions there are many other questions regarding the content and format of access logs that would have to be commonly agreed, e.g. how fine grained the logging should be, how detailed the descriptions should be etc. Failing in this area could make the result very confusing and even misleading for citizens. Badly implemented, the result might even do more harm than good. Therefore, instead of talking about technical details it would be better to concentrate on the targeted outcome from citizen’s point of view first.

Back to the topic

Back to the earlier question – could the message log be used for providing access to the information regarding who has accessed my data and when? Basically yes, but a partial solution would not bring very much value to citizens. In addition, development of new features would be required to remove the manual work regarding handling of the logs. In practice, the message log alone is not a sufficient solution, but it can definitely be used as a part of a wider solution discussed before. Therefore, the message log alone cannot be used for providing access to the information regarding who has accessed my data and when.

That’s it about the X-Road logs for now. There is more to come later...