There’s No Distributed Ledger Technology (DLT) in X-Road

In 2018 I wrote a blog post about blockchain and X-Road. The aim of the post was to explain that there's no blockchain technology in X-Road, unlike multiple writings had been claiming. Despite the facts provided in the post, still today, X-Road is often called a blockchain or distributed ledger technology (DLT) based solution. X-Road has experienced several major changes since 2018, but its relationship to blockchain hasn’t changed – there’s still no blockchain technology in X-Road.

My earlier blog post already explained the difference between X-Road and blockchain. But what is DLT, and how does it differ from blockchain? And most importantly, what’s the relationship between DLT and X-Road? Let’s find out!

Distributed ledgers

Essentially, DLT or a distributed ledger is a distributed database spread across multiple nodes that all hold an identical copy of the data stored in the ledger. The data is replicated, shared, and synchronized across all the nodes, and the nodes update themselves independently. When the ledger is updated, the nodes use a consensus algorithm to agree on which copy is correct. Once the nodes reach a consensus, they all update themselves with the new copy. In other words, the consensus algorithm enables the nodes to agree on the state of the ledger.

Typically, distributed ledgers are decentralized since they are based on a distributed architecture and don't have centralized governance. However, there are distributed ledgers that have some level of central administration. Therefore, it can be argued not all the distributed ledgers are entirely decentralized.

A distributed ledger may be permissionless, permissioned, or hybrid. Permissionless ledgers are public, and everyone can participate in the network and access the data stored in the ledger. Instead, permissioned ledgers are private, and they only allow trusted nodes to join the network. In private ledgers, the information is available to a specific group of permitted participants, and a network operator decides who is allowed to join the network. In addition, there are also hybrid ledgers that are a combination of permissionless and permissioned networks, e.g., anyone can read data from the ledger, but only authorized nodes are allowed to write data to the ledger.

Distributed ledgers typically have several properties, including but not limited to immutability, transparency, and trust. All the transactions in the ledger are immutable since once a transaction exists in the ledger, nobody can change it. Distributed ledgers are transparent because all the nodes have the same copy of the data. Also, distributed ledgers enable direct trust between parties that don't necessarily trust each other. Immutability and trust are accomplished through cryptography and digital signatures.

There are various types of distributed ledgers that implement the properties differently and have their characteristics. Different kinds of ledgers include blockchain, Directed Acyclic Graph (DAG), Hashgraph, and Holochain.

Sometimes distributed ledger or DLT and blockchain are used as synonyms. However, as we have just seen, there are different types of distributed ledgers, and blockchain is just one type of distributed ledger. Blockchain is a data structure of blocks linked to one another via cryptographic hashes, but distributed ledgers don’t require such a structure or chain.

In general, blockchains are distributed ledgers, but not all distributed ledgers are blockchains. Therefore, distributed ledger and blockchain shouldn’t be used as synonyms.

Is X-Road a distributed ledger?

X-Road and distributed ledgers are both based on decentralized architecture. Distributed ledgers don't typically have central administration, which means that they're entirely decentralized. Instead, X-Road is technically decentralized, but an X-Road ecosystem is always built around a centralized governance model managed by the X-Road operator.

A fundamental difference between X-Road and distributed ledgers is that distributed ledgers are decentralized databases for storing data, while X-Road is a decentralized solution to exchange data. All the distributed ledger nodes hold the same copy of the data, and they agree on the latest version using a consensus algorithm. Instead, in X-Road, data is exchanged between two data exchange parties, and other parties do not have access to it. Also, X-Road nodes do not permanently store data - they only act as secure proxies. All the nodes of an X-Road network have their own sequence of transactions that are not shared with any other nodes – not even in a Security Server cluster. Therefore, there’s no need for a consensus algorithm in X-Road.

Based on the above reasoning, it's evident that X-Road is not a distributed ledger. Comparing X-Road and distributed ledgers is like comparing apples to oranges. However, under the hood, both distributed ledgers and X-Road use the same mechanisms, cryptography and digital signatures, to secure their operations – they’re just applied differently.

Is distributed ledger technology used in X-Road?

Distributed ledger technology is not currently used in any areas of X-Road. The data exchange always happens between two parties, and the data is not shared with other parties of the X-Road ecosystem. Even in a Security Server cluster with an external load balancer where all the nodes share the same identity, the exchanged messages are Security Server specific. Only configuration is shared between the nodes. Consequently, the logs produced by different nodes of the cluster are unique since each message is processed by one node in the cluster only. Also, the logs are stored on the nodes only temporarily - once they’ve been archived, they are removed. Therefore, there aren't currently features that would directly benefit from distributed ledgers when it comes to data exchange. Although, that doesn't mean that some additional features that aren't directly related to data exchange couldn't be implemented using distributed ledgers.

However, some other aspects of X-Road could potentially benefit from distributed ledger technology in the future. For example, how configuration details are managed on the Central Server and distributed to the X-Road ecosystem through global configuration is a potential topic to study further. Also, when thinking about X-Road’s trust model and how organizational identities of X-Road members are managed, self-sovereign identity (SSI) together with decentralized identifiers (DIDs) and verifiable credentials could provide some exciting alternatives in the future. But that’s a topic for another blog post.

All in all, X-Road’s isn’t currently based on distributed ledger technology nor blockchain. When it comes to using distributed ledger technology in X-Road, even if something can be done using distributed ledger technology, it doesn't automatically mean that it should be used. A common principle in the X-Road development is to choose the technology that’s the best fit for the job instead of selecting the most popular alternative.

Protecting Data at Rest in X-Road 7

One of the core features of X-Road is to protect data in transit, and X-Road includes several mechanisms to protect data when it travels between the data exchange parties. Even if X-Road is not a data storage solution, it doesn’t mean that the Security Server wouldn’t store any sensitive data, at least temporarily. Therefore, X-Road 7.0.0 will introduce new security features to protect data at rest on the Security Server. The new features are:

• encrypt backup files (opt-in)

• verify the integrity of backup files before running a restore

• encrypted message payload in message log database (opt-in)

• encrypted message log archives (opt-in)

• group message log archives by member or subsystem (opt-in).

 This blog post gives an overview of the new security features and their key characteristics.

Common design principles

The new data protection features provide the possibility to encrypt and sign specific data stored on the Security Server. By default, encryption is turned off, and the Security Server administrator can manually enable it. All the encryption-related configuration operations require shell access to the Security Server.

The new data protection features have been implemented using the GNU Privacy Guard (GnuPG) – an open-source implementation of the OpenPGP standard as defined by RFC4880. All operations related to backup files and message log archive files have been implemented in an external GnuPG process. Instead, the message log database encryption has been implemented in Java code.

Image 1. Encryption keys that are used to protect data at rest on the Security Server.

The required keys are stored in GPG keyrings, and Security Server's internal key (1) used by default is generated automatically. The Security Server administrators can manually configure additional and member-specific keys (2, 3, 4). All the new encryption keys are illustrated in image 1.

Configuration backup encryption

Security Server configuration backups contain sensitive information, such as database connection credentials, private keys, etc., and therefore, they should be protected from unauthorized access. Starting from version 7.0.0, the Security Server supports signing and encrypting the backup files.

The backups are always signed, but backup encryption is initially disabled. When encryption is enabled, backup files are encrypted when they are created and stored on disk encrypted. It applies to both automatically created and user-created backups. When it comes to user-created backups, the configuration applies to Security Server UI and REST management API.

When backup encryption is enabled, the backups are signed and encrypted using the Security Server’s automatically generated internal key pair (1). In addition, the Security Server administrator can upload additional keys (2) to the Security Server that are used to encrypt the backups together with the internal key. In practice, the backups are encrypted with the internal key and all additional keys. The backups are always signed with the internal key.

During restore, the Security Server verifies the consistency of backup files automatically. However, if the Security Server’s internal key pair is lost, automatic verification is no longer possible. In that case, the Security Server administrator can restore the decrypted backups from the command line with appropriate options that enable skipping integrity verification.

Configuring additional encryption keys is strongly recommended since the Security Server administrator can use them to decrypt configuration backup files in a situation where the internal key pair is lost. Without additional keys, the encrypted configuration backups are unusable in such a situation.

Message log archive grouping and encryption

Message log archive files may contain sensitive information that different organizations can own if multiple members share a single Security Server. Therefore, in version 7, message log archive files can be encrypted and grouped by member or subsystem.

Message log archive encryption and grouping can be configured separately. For example, the archives can be encrypted but not grouped (or vice versa). By default, both features are disabled. Grouping and encryption are enabled/disabled on a Security Server level - they are either enabled or disabled for all the members and subsystems. It's not possible to enable/disable neither of them for selected members nor subsystems only. When encryption and grouping are combined, each member's messages are stored in separate files encrypted with member-specific keys (4). In addition, when message log archive encryption is enabled, the signed document download service returns log records encrypted.

Message grouping can be configured by a member or subsystem. By default, grouping is disabled, and all archive files go to the same default group. Message log archive files that belong to the same group are chained using cryptographic hashes. The implementation is not based on blockchain. However, there may be holes in the chains if grouping is turned on and off or the grouping level (none/member/subsystem) changes. Every time grouping is turned on/off or the grouping level changes, the chains start from zero.

When message log archive encryption is enabled, the archive files are encrypted and signed using the Security Server’s internal key (1) by default. Instead, when archive encryption and grouping are combined, the Security Server administrator must configure one or more member-specific encryption keys (4) for each member. The member-specific keys are used to encrypt the archive files belonging to the member. If the Security Server administrator configures no member-specific key, the Security Server's internal key is used as a fallback.

Also, the format of the message log archives' filenames depends on the configuration of the Security Server. The format varies between encrypted and plain text archives and between different grouping levels. If the archive files are automatically processed based on the file name, the matching rules may have to be adjusted. More information about the filename format is available in the Security Server user guide.

Message log database encryption

Just like the message log archive files, the message log database may contain sensitive information. Starting from version 7.0.0, the message body stored in the message log database can be optionally encrypted, but the default is no encryption. The database encryption is fully transparent to all the external interfaces, e.g., the signed document download service. The Security Server administrator is responsible for configuring the database encryption key (3) before enabling database encryption.

In the message log database, there are two separate columns for plaintext ("message") and encrypted ("ciphermessage") message body. The message body is always stored in one of the two columns depending on the configuration. Instead, the other column that is not used is left empty. When message log database encryption is enabled/disabled, the change doesn't affect already existing records in the database. For example, when message log database encryption is enabled, all the records created after the configuration change will be encrypted and stored in the "ciphermessage" column. Instead, all the records stored before the change will remain in plaintext in the "message" column.

Thanks to the way how the message body encryption in the message log database has been implemented, the change doesn't affect any external integrations. For example, suppose message logs are exported directly from the message log database to some centralized logging system. In that case, the integration doesn't need to be changed if the message log database encryption continues to be disabled.

During message log archiving, the message body is decrypted and written to the archive file in plaintext. Message log archive encryption is configured separately, and when it's enabled, the whole archive file is signed and encrypted. In practice, the message log database encryption may be enabled while the message log archive encryption is disabled (or vice versa). However, if the message log contains sensitive information, it's recommended to enable both the message log database encryption and the message log archive encryption.

Future enhancements

The new data protection features introduced in version 7.0.0 will be developed further in the upcoming releases. For example, one of the potential enhancements is to provide information on the statutes of different encryption configuration options in the Security Server UI to detect the current state and possible configuration errors. Also, making various encryption-related configuration options available through the Security Server UI and REST management API is part of the future development plans.

We’re glad to receive feedback about the new features and contributions regarding further enhancements.

Message Logging in X-Road 7

One of X-Road’s key features is to provide undeniable and tamper-proof evidence of data exchange. X-Road does this by digitally signing, timestamping, and logging all the messages processed by the Security Server. The logs can be used in a court proceeding as evidence. This logging feature is known as the message log. These previous blog posts provide additional information about the message log:

• X-Road Logs Explained - Part 1

• X-Road Logs Explained - Part 2

• X-Road Logs Explained - Part 3

Depending on the data exchange use case, indisputable proof that can be used as legal evidence may be required. On the other hand, regular application logs provide enough information in many cases, and no additional guarantees are needed. In the latter case, X-Road message logs may not be required, or it's enough to log only the metadata. To better support different kinds of use cases and logging requirements, message logging has been made more configurable in X-Road 7. This blog post provides more information about different configuration alternatives available in X-Road 7.

Supported alternatives in X-Road 7

Starting from X-Road 7.0.0, the Security Server supports three options for configuring message logging:

• Full logging

  • The whole message, including both message body and metadata, is logged.

  • The log records can be verified afterward, and they can be used as evidence.

  • Timestamping service is required.

• Metadata logging

  • Only metadata which includes X-Road-specific message headers is logged while the message body is not logged.

  • Verifying the log records afterward is not possible, and they cannot be used as evidence.

  • Timestamping service is required.

• No message logging

  • Message logging is fully disabled. Neither message body nor metadata is logged.

  • No log records are generated.

  • Timestamping service is not required.

Regardless of how logging is configured, messages exchanged between Security Servers are always signed and encrypted. Also, when full logging or metadata logging is enabled, the Security Server produces a signed and timestamped document (Associated Signature Container - ASiC) for each request and response. 

Is it possible to combine different alternatives? 

Full logging and metadata logging can be configured on Security Server and subsystem level. When the Security Server level configuration is used, the same configuration is applied to all the subsystems. Instead, when the subsystem level configuration is used, the configuration is applied to specific subsystems only. In addition, combining the Security Server and subsystem level configurations is also possible, e.g., set metadata logging on the Security Server level and enable full logging for specific subsystems only (or vice versa). Instead, message logging is fully disabled on a Security Server level and it cannot be combined with the other alternatives. Therefore, a subsystem that requires full or metadata logging should not be registered on the Same Security server with a subsystem that requires entirely disabling message logging.

Which alternative should I use?

Multiple factors should be considered when selecting the logging configuration for a data exchange use case. First, there may be legal requirements regarding logging and retention period of the logs. Second,  some logging-related requirements that all the member organisations must follow may have been set by the X-Road operator. Third, there may be some organization-specific requirements for different kinds of use cases. Fourth, the other data exchange party of a specific data exchange use case may have requirements for logging. It's essential to be aware of the different requirements and configure logging accordingly.

To make the decision easier for member organisations, X-Road operators may define a default value for logging-related configuration in a country-specific meta-package. Without any country-specific configuration, the Security Server uses full logging by default. For example, suppose the operator requires using metadata logging by default. In that case, the operator can set the configuration in the country-specific meta-package so that all member organisations use metadata logging by default. The configuration must be updated manually only if some other logging alternative should be used. However, fully disabling message logging is not possible using country-specific meta-packages. Instead, it must be configured separately by the Security Server administrator.

How does the logging configuration affect the Security Server in general?

The logging configuration has some direct and indirect effects on the other features of the Security Server. Fully disabling logging increases the overall performance, reduces disk space consumption, and doesn't require a timestamping service to be configured. Also, moving the message log archives to external long-term storage is not needed since, without the message logging, there are no archive files either. Overall, fully disabling message logging improves the performance and decreases the environmental footprint of the Security Server. If logging is disabled in the whole X-Road ecosystem, running the ecosystem without a timestamping service is possible. However, even if performance and environmental reasons are essential, legal and administrative requirements must not be forgotten.

All the changes related to the message log encryption and archive grouping will be covered in a separate blog post together with other new encryption features.

Configuration Changes in X-Road 7

X-Road 7 will be backward compatible on the interface level with X-Road 6 since there are no changes in the Message Protocol for SOAP and the Message Protocol for REST. Upgrading the Security Server software is enough, and no changes to the connected information systems are required. However, the Service Metadata Protocol for REST and the Security Server REST management API have some changes that might not be fully backward compatible depending on the client application. Also, some Security Server's configuration changes are not backward compatible, and manual actions may be necessary depending on the current configuration. This blog post provides more information about the backward-incompatible changes in the configuration.

A new configuration file for overrides

X-Road 7 will change how some configuration overrides are handled by X-Road system services, e.g., proxy, signer, etc. The changes make the configuration more secure and more consistent. Besides the Security Server, the changes affect the Central Server and Configuration Proxy too.

In version 6, the "/etc/xroad/services/local.conf" file is used to override the environment variables for X-Road system services, e.g., memory allocation for proxy and signer. The problem with the implementation is that the configuration file is an executable that may also contain scripts and code in addition to the configuration overrides. Since the file is executed during restore, it enables code execution as a part of the restore process.

In X-Road 7, the configuration override mechanism will be changed to define the overrides in a non-executable properties file. Starting from X-Road 7.0.0., the properties file "/etc/xroad/services/local.properties" is used to override X-Road's environment variables. The file may contain configuration overrides only to predefined environment variables, and everything else is ignored.

Changes in environment variable names

Also, the format and the names of X-Road's environment variables will be different. Starting from version 7.0.0, all the variables are prefixed with "XROAD_". For example, the difference in customising memory allocation for proxy in X-Road 6 and X-Road 7:

X-Road 6

File "/etc/xroad/services/local.conf":

PROXY_PARAMS="$PROXY_PARAMS -Xms200m -Xmx2000m "

X-Road 7

File "/etc/xroad/services/local.properties":

XROAD_PROXY_PARAMS=-Xms200m -Xmx2000m

Here's a list of the X-Road-related environment variables and their names in different X-Road versions.

X-Road 6	X-Road 7
XROAD_PARAMS	XROAD_PARAMS
SIGNER_PARAMS	XROAD_SIGNER_PARAMS
ADDON_PARAMS	XROAD_ADDON_PARAMS
CONFCLIENT_PARAMS	XROAD_CONFCLIENT_PARAMS
CONFPROXY_PARAMS	XROAD_CONFPROXY_PARAMS
JETTY_PARAMS	XROAD_JETTY_PARAMS
MONITOR_PARAMS	XROAD_MONITOR_PARAMS
OPMON_PARAMS	XROAD_OPMON_PARAMS
PROXY_PARAMS	XROAD_PROXY_PARAMS
PROXY_UI_API_PARAMS	XROAD_PROXY_UI_API_PARAMS
SIGNER_CONSOLE_PARAMS	XROAD_SIGNER_CONSOLE_PARAMS

Other changes that affect configuration

In addition, there are some changes in the backup and restore features that affect the configuration too. Permissions for all the restored files are set to safe, constant values during a restore which means that permissions in the backup file are ignored. Also, the Security Server database backup format has changed, and the new format prevents the execution of arbitrary commands during restore. All the changes related to the Security Server backup encryption will be covered in a separate blog post together with other new encryption features.

What happens when I upgrade to X-Road 7?

Unfortunately, the changes on the Security Server are backward incompatible. It means that backup files generated on X-Road 6 Security Server cannot be restored on X-Road 7 Security Server. However, the changes on the Central Server are backward compatible.

Despite the changes, when upgrading from version 6 to 7, all overrides defined in "/etc/xroad/services/local.conf" will continue to work as before, including the old environmental variable names. However, starting from X-Road 7.0.0, the backups no longer contain the "/etc/xroad/services/local.conf" file, but instead the "/etc/xroad/services/local.properties" file will be included. It means that in case a Central Server or Security Server is restored from a backup, all local configuration overrides defined in "/etc/xroad/services/local.conf" are lost.

Also, in version 7, the backups no longer contain other ".conf" files located in the "/etc/xroad/services" directory. Those files are system configuration files that should never be manually modified since X-Road version upgrades always overwrite them. In case they're lost or corrupted, it's enough to re-install the X-Road packages. However, if the files have been manually modified, all the modifications are lost in case of restore.

It is strongly recommended to migrate all configuration overrides to the new "/etc/xroad/services/local.properties" file when upgrading from version 6 to 7. There's no automation in the migration, and therefore, migrating the configuration overrides must be done manually by the administrator. However, if the "/etc/xroad/services/local.conf" configuration file isn't used in X-Road 6, no actions are required.The migration includes changing the variable names to the new "XROAD_" prefixed format and updating the values according to the new properties format. In case the old "/etc/xroad/services/local.conf" configuration file contains something other than modifications to X-Road-related environmental variables, the changes must be handled in some other way. The new configuration override mechanism doesn't support additional configuration changes.

When both old and new configuration files exist

Also, the new and old configuration files can coexist side by side. When both "local.conf" and "local.properties" files exist, "local.properties" is applied first, and "local.conf" is applied second. However, "local.properties" can only append to the new variables ("XROAD_*"), while "local.conf" can change any parts of the configuration. As a result, values defined in "local.conf" overwrite values defined in "local.properties" if the same property is defined in both files. However, non-overlapping values defined in "local.properties" are still applied. Also, in case both old and new variable names (e.g., "SIGNER_PARAMS", "XROAD_SIGNER_PARAMS") are used in "local.conf", the new variable is applied. For example:

When "/etc/xroad/services/local.properties" contains:

XROAD_SIGNER_PARAMS=-Dprop1=newValue1 -Dprop2=newValue2

And "/etc/xroad/services/local.conf" contains:

SIGNER_PARAMS="$SIGNER_PARAMS -Dlegacy=true"

XROAD_SIGNER_PARAMS="$XROAD_SIGNER_PARAMS -Dprop1=oldValue1   -Dlegacy=false"

The resulting values for different "XROAD_SIGNER_PARAMS" related properties are:

• prop1=oldValue1

• prop2=newValue2

• legacy=false

More detailed instructions for the migration are provided in the X-Road 7 Migration Guide.

Get Ready for X-Road 7

The long-awaited X-Road 7 “Unicorn” is almost here. Building the new version started in October 2020, so it has been rather a marathon than a sprint. The first public beta version of the Security Server was released in December 2020, and it provided the first view on the new major version. The first public release introduced a new visual style, but it didn't offer any functional changes yet. Since the initial release, many things have changed under the hood, and the new visual style has been polished. Now it's an excellent moment to look into the mouth of the Unicorn and have a comprehensive overview of X-Road 7.

X-Road 6 was initially released in 2015, and it has experienced several major changes during its lifecycle, but the core principles have remained the same. Over the years, X-Road 6 has proven to be secure, reliable, and scalable. Therefore, it was decided to use it as a basis for the next major version. Thanks to the solid foundation, there’s no need to reinvent the wheel. The aim is to keep all the good in X-Road 6 and get rid of the bad and the ugly. In other words, X-Road 7 will have all the strengths of X-Road 6 with numerous improvements in various areas. All in all, X-Road 7 is an evolutionary version of X-Road 6.

What happens to X-Road 6 when X-Road 7 is released?

X-Road’s current lifecycle policy is to support the latest version plus two previous versions. The supported versions are defined on MAJOR.MINOR level so the release of patch versions (MAJOR.MINOR.PATCH) does not effect on the support. The supported versions receive patches in case of bugs and vulnerabilities. Also, version upgrades are supported and tested between the supported versions. However, with X-Road 7.0.0, there's one exception – upgrade to version 7.0.0 is supported from version 6.26.0 only. If you're running an older version of X-Road, upgrade to version 6.26.0 is required first before upgrading to version 7.0.0.

Version 6.26.0 is the last planned release for X-Road 6. According to X-Road’s lifecycle policy, it will be supported until version 7.2.0 is released. Currently, the planned release date for version 7.2.0 is November 2022. However, version 6.26.0 or any other older version doesn't stop working when a new X-Road version is released. Instead, older versions continue to work without any immediate effects.

Nevertheless, it’s strongly recommended to use a supported version of X-Road to be safe if bugs or security vulnerabilities are detected. In addition, X-Road member organisations should follow X-Road ecosystem-specific guidelines and policies regarding supported versions set by the X-Road operator. For example, a common practice is that the operator first upgrades the Central Server to the new version, and the member organisations are allowed to upgrade their Security Servers only after that. 

Is X-Road 7 backward compatible with X-Road 6?

The last major version upgrade from X-Road 5 to X-Road 6 required a lot of work from X-Road member organisations because the message protocol between information systems and the Security Server changed. It meant that upgrading the Security Server wasn't enough, but the information systems required changes too. The good news is that X-Road 7 is backward compatible on the interface level with X-Road 6 since there are no changes in the Message Protocol for SOAP and the Message Protocol for REST. Upgrading the Security Server software is enough, and no changes to the connected information systems are required. However, the Service Metadata Protocol for REST and the Security Server REST management API have some changes that might not be fully backward compatible depending on the client application.

However, some changes in the Security Server's configuration are not backward compatible, and manual actions may be required depending on the current configuration. Overall, the estimated effort for the activities is low, and the version upgrade from version 6 to version 7 is more like a version upgrade between two X-Road 6 minor versions, e.g., version upgrade from version 6.23.0 to version 6.24.0.

How is X-Road 7 developed?

X-Road 7 will be implemented iteratively using agile software development methods. All the changes will not be included in the first production version, but they will be introduced one by one over time in various X-Road 7 minor versions.

The development of X-Road 7 is divided into multiple high-level focus areas. Each focus area consists of several topics that will be turned into actual features. The high-level focus areas and their main topics are:

• messaging patterns

• message logs

• onboarding process

• architecture

• operational insights

• sustainability.

More information on the focus areas is available on the X-Road website.

What’s new in X-Road 7.0.0? 

In version 7.0.0, the focus is on the Security Server, and there are only a few minor changes on the Central Server side. The Security Server will have a new visual style that implements the X-Road 7 visual style guide, while the Central Server still has the version 6 visual style. Here’s a summary of changes included in version 7.0.0:

• New X-Road 7 look and feel for the Security Server UI.

• Security improvements on the Security Server:

  • Encrypt backup files (opt-in)

  • Verify integrity of backup files on restore.

• Improvements in Security Server message logging:

  • Encrypt message payload in message log database (opt-in)

  • Encrypt message log archives (opt-in)

  • Group message log archives by member or subsystem (opt-in)

  • Support for fully disabling message logging.

• Change PIN code on the Security Server.

• Return REST API type (OPENAPI3 / REST) and API endpoints in REST metaservice responses.

• Run the Security Server on Java 11 by default.

• Make Security Server more modular by enabling installation without a local Postgres server.

• Version compatibility check for version upgrades - it is no longer possible to update from an unsupported version.

• Official Docker support for the Security Server with the Security Server Sidecar images.

• Other enhancements and bug fixes.

The development focus will shift to the Central Server in 2022 when it gets a new user interface and a management REST API. Also, the new Central Server will include several architectural changes, which will make X-Road more extensible and easier to operate.

What is the release schedule?

Version 7.0.0 will be released in Q4 / 2021. Before the final release version, there will be a public beta version available in October 2021. The beta version will include all the main features available in the final release, and it's targeted at anyone interested in testing the new version in advance. The official release notes that provide detailed information about all the changes included in the release will be published together with the beta. In addition, a separate migration guide that offers detailed information about any manual actions required in the upgrade process will be published.

This writing is the first part in a series of X-Road 7 related blog posts. More writings providing insights on X-Road 7 will be published on the NIIS blog during the following months. Stay tuned!

Reducing the environmental impact of X-Road

Introduction 

Nordic Institute for Interoperability Solutions (NIIS) has set a long-term goal to make X-Road the most sustainable data exchange solution in the world. To better understand the direct and tangible environmental impacts of the use of X-Road software, NIIS commissioned research to assess the current emissions profile across X-Road’s operations and services. In doing so, it delivers targeted recommendations for emissions reduction and sustainable business practices that may be integrated into future decision making.

The study was performed by Gofore and Stockholm Environment Institute Tallinn (SEI) in close collaboration with NIIS and the X-Road Governing Authorities in Estonia and Finland. Together, experts built an emissions calculator to assess the emissions profile of the X-Road instance, subject to the emissions boundary defined in our first blog post. The calculations used in this process have been derived from peer-reviewed literature and tailored to X-Road use cases in Estonia and Finland, which is well described in our previous blog post. However, the emission calculator is comprehensive enough to perform calculations for any instance residing in any region.

Results

The results of the calculator give a clear picture of the total carbon footprint of X-Road, as well as its individual components. The key findings demonstrate: 

1. Around 96 % of total emissions are related to the operations of X-Road Security Servers. 

2. Data transmission and storage provide marginal contributions to the total carbon footprint (around 1 % and 3 %, respectively).

3. The annual carbon footprint for Estonia and Finland was approximated as 45,685 KgCO2e and 22,593 KgCO2e, respectively.

4. The discrepancy between the two countries relates mostly to the difference in average electricity grid emission factor – this reflects the larger percentage of low carbon electricity sources in the Finnish grid and the relative reliance in Estonia on shale oil.

Based on these results, we have derived recommendations to manage or mitigate emissions relevant to different stakeholder groups. These groups have been determined based on their different levels of access and influence over the way X-Road is used; including X-Road user organisations (X-Road members), the X-Road governing authorities overseeing regulations guiding the service application, NIIS as the product owner and developer of X-Road, and the broader public. The recommendations can briefly be summarised as follows:

1. The future development of X-Road could ensure flexibility for stakeholders to disable or reduce the emissions of components, such as message logging and timestamping, subject to potential performance and security requirements.

2. Wherever possible, NIIS can recommend an energy tracking application to enable emissions modelling of X-Road in real-time. This will integrate transparent monitoring of emissions in reporting and encourage users to evaluate ways to reduce emissions as they will have a reference (Business As Usual) data set to experiment with and understand their emissions footprint.

3. X-Road governing authorities can provide clear information and support emissions reduction strategies, such as granting permission to host servers on the public cloud and reducing mandatory requirements for message logs and/or timestamping.

4. X-Road members should implement X-Road services efficiently and in a manner optimised for emissions reduction from the outset. This could include ensuring that equipment is efficient and server utilisation is maximised. Alternatively, servers could be hosted on the public cloud – as it is beginning to be allowed for organisations requiring the highest level of security, such as UK Defence – if permitted by local regulations and the governing authority.

5. Where the option exists, infrastructure should be powered using renewable electricity. Moreover, best practice should be followed, such as using power-efficient hardware devices and optimised data compression. 

Process of the study

As discussed in previous blog posts, this study was divided into three phases, with results and methods published for feedback by a steering committee and technical experts after each phase. The phases were as follows:

1. Determining an emissions boundary and mapping the main causes of environmental impacts of the X-Road instance.

2. Building a Carbon Footprint calculator for X-Road, based on best practice and X-Road use cases.

3. Defining recommendations for improving the sustainability of X-Road.

The calculator was designed to ensure a high degree of flexibility while obtaining directional results that could be accepted with high confidence. This ensured validity across the instances in Finland and Estonia and across the varying circumstances and technical literacy of different X-Road users. 

Outcomes

The report is available here and the simplified X-Road Emissions Calculator is available here.

The X-Road Carbon Emissions Calculator – Methodology and Results

Introduction 

To support Nordic Institute for Interoperability Solution’s (NIIS) mission on making X-Road the ‘most sustainable data exchange solution in the world’, a study is being carried out by Gofore and Stockholm Environment Institute Tallinn to assess the carbon impacts of the solution. In the first stage we defined the scope of the project and described what would and wouldn’t be included. We have proceeded a step further and developed a unique emissions calculator that measures the carbon footprint of X-Road’s operations.  The calculator was built using data from instances in Estonia and Finland but is comprehensive enough to perform calculations for any instance residing in any region.

From energy to emissions

The methodology proceeded in two stages. First, the team developed a model to understand the electricity burden of operating an X-Road Security Server. Once the electricity consumption of one Security Server was determined, the results were multiplied with the total number of Security Servers in an entire X-Road ecosystem. Next, this was converted into an estimation of released emissions by multiplying this value by an ‘emission factor’ for grid electricity. The emission factor describes the emissions for each unit of electricity and considers all the different sources of electricity generation within a country or region. This is important as the content of this ‘electricity mix’ can be wildly different between different countries, leading to very different emissions outcomes for the same energy use. Results are reported in units of CO2 equivalent (CO2e), which includes other important greenhouse gases, such as methane. For these calculations, emission factors published by Association of Issuing Bodies (AIB) for 2019 are used. 

The model

This model was refined in subsequent steps based on new literature findings, updated assumptions testing among the internal team, and interviews with end-users and industry experts. As it is not possible to investigate every single source of emissions within a single project, it is important to focus on the largest that will define the size of the carbon footprint. As such, three main sources of emission were identified:  infrastructure, data transaction and data storage. In this context, the ‘infrastructure’ component only included the Security Server which is required to process data and enable secure data exchange. The team hypothesized this would be the main source of emissions, since multiple processors can collectively contribute vast amounts of heat emission and energy consumption. To compensate, data storage was considered as a separate component to account for the recording of all the transactions occurring over X-Road servers.  

Infrastructure

The main physical infrastructure that enables secure data exchange through X-Road is the Security Server. For simplification, only CPU and RAM are considered as these components are responsible for almost all the energy consumption in a security server.  A Fujitsu server with a model “FUJITSU Server PRIMERGY RX1330 M4 is assumed to be a standard server throughout the calculations. This serves as a good representation of a sample server employed across the two instances based on surveys conducted with the Finnish and Estonian authorities. The server employs an Intel® Xeon® E-2288G Processor with 16M Cache, 3.70 GHz and 8 cores, and was published in the database in 2019. Values for the energy use of this server were obtained from a widely used published database, SPECpower_ssj2008.

The processor’s energy consumption can be modelled as a directly proportional relationship between CPU utilization and Average Active Power. In turn, the RAM energy consumption model is based on a study conducted by Pedro H. P. Castro et al. The RAM’s power consumption is divided into background power (depends only on memory states and on the frequency of the operation) and operational power (product of memory bandwidth and the power required to run a particular command). The total power consumption was taken as the sum of these two individual components.

A final factor to consider is the energy consumption of the supporting infrastructure. This is widely considered in the industry through a factor known as the power use effectiveness (PUE). PUE describes the amount of energy used by the IT devices compared to the amount used by the supporting infrastructure (in this project a PUE of 1.58 was used). Thus, the total energy consumption of a Security Server is calculated as a product of total electricity consumption and PUE.

Transaction

Work by Aslan et al pointed to an average data transmission efficiency of 0.06 kWh /GB for fixed line transmission, a value that was also seen empirically to half every 2 years. For 2021, including the expected efficiency gains, this value is found to be 0.0075 kWh of electricity consumed for 1 GB of data transferred over the local internet. The methodology simply involves taking the product of the total amount of data exchanged over X-Road and the electricity consumption per GB of data transferred. 

Storage

According to bilateral exchange of information with different X-Road members, data is stored in hard disk drives (HDDs). A study conducted by Adam Lewis et al from Athens state university outlines a comprehensive approach that enables accurate energy consumption calculations of data storage in an HDD. For calculating the energy consumption of data storage, an HDD’s start up power, power consumption during data writing, power consumption in idle mode and finally, power consumption in standby mode is all calculated and summed.  

Carbon footprint results

The following table summarizes the total emissions of the Estonian and Finnish instances.

Table 1. Estimated carbon footprint summary of X-Road operations in Estonia and Finland.

The total carbon dioxide equivalent (CO2e) emissions for Estonia amount up to 45,685 kgs and for Finland up to 22,593 kgs. The results include emissions from all three main operations (i.e.,  infrastructure,  transaction and storage). Although the Finnish instance has a lot more Security Servers than the Estonian instance (173 in Estonia and 290 in Finland), the disparity in the electricity emission factor (0.723 kg of CO2e/kWh for Estonia and 0.136 kg of CO2e/kWh for Finland) is the key reason for the contrast between the aggregated emission amounts of both regions. A comparison between the emissions of the three main operations allows us to identify and analyse areas which contribute most significantly to the total emissions. Resulting emissions from the electricity consumption are depicted in figure below.

Figure 1. Emission by main operations in the Finnish and Estonian instances

The electricity consumption (and therefore emissions) by the Security Servers in comparison to data transaction and storage is substantially high and dominates the total share. Emissions from servers contribute to 96% of the total emissions which amount to 21,617 kgs of CO2e for the Finnish instance and 43,999 kgs of CO2e for the Estonian instance. This directly points to the primary place of focus in order to make X-Road’s operations more sustainable. Data storage contributes to a mere 3% that amount to 1,507 kgs of CO2e for Estonia and 950 kgs of CO2e for Finland. The remaining are from data transaction (less than 1%) for both instances. 

In conclusion, the calculator depicts a holistic view of the total emissions due to X-Road’s operations. For the scope of this project, calculations are done for Estonia and Finland though the calculator is configurable for any instance in any region. This methodology aims to cater emission calculations to the key areas which emerge to have the greatest emissions, and to be flexible to the different circumstances used by X-road members. As there are hundreds of different Security Servers and storage devices spread all across Finland and Estonia, it is impossible to track down exact models of the components with their corresponding power consumption specifications and data exchange numbers. We therefore hope our approach strikes the ideal balance between accuracy and flexibility.

We welcome your feedback on the methodology and results and will shortly publish more details in a formal report. The next stage of the project will develop a series of recommendations for NIIS and X-Road members to ensure carbon emissions are minimized without compromising the effectiveness of the X-Road service. 

---

Adil Aslam is a Junior Expert At SEI Tallinn and joined the Climate and Energy programme in November 2020.

Adil has diverse experience in manufacturing, energy consultancy, modelling and project management. He is an expert in conventional and renewable energy systems while being proficient in a whole range of energy simulation software. He is adept in programming and has used Python and Matlab to develop various energy models. His interests lie in energy systems, sector coupling, green finance, sustainable business models, electricity markets and simulations.

He is currently working on simulating hydrogen potential in Estonia, climate neutral scenarios for 2050 and providing his expertise in various energy related projects. He actively takes part in the company’s business development operations and aims to explore different business opportunties.

He graduated from Technische Universität Berlin, Germany in 2020 with a Master’s in Business Engineering Energy. He did his Master’s thesis at Forschungszentrum Julich (one of the largest interdisciplinary research centres in Europe) and the Institut für Stromrichtertechnik und Elektrische Antriebe (ISEA) RWTH, where he worked on modelling an energy management system for residential PV home systems to sustain blackouts.

Introducing X-Road Emissions Calculator Project

Introduction

NIIS has set an ambitious goal to make X-Road the 'most sustainable data exchange layer solution in the world' and thereby align its operational model with the climate and sustainable development goals articulated under the Paris Agreement and the 2030 agenda for sustainable development. That is why X-Road's environmental impact is being identified by NIIS, Gofore and the Stockholm Environment Institute (SEI).

The study is being carried out by a team, which is a combination of the sectoral experience provided by Gofore and the academic competence provided by SEI Tallinn. Gofore's experts have experience with X-Road core development processes, X-Road implementations, and X-Road service developments. SEI Tallinn is a leading sustainability think tank ranked #2 globally in the environmental field by Uni Penn ranking. SEI Tallinn has long-standing experience in climate and energy policy scenarios, carbon footprint calculations and life-cycle impact assessments.

The current project measures the carbon footprint of X-Road operations within Estonia and Finland. It will give recommendations for all related parties, including X-Road governing authorities and user organisations around the world.

Project is in its early stages. The partners have scoped the most important environmental impacts of X-Road instance and outlined the initial approach for X-Road carbon footprint calculation that is expected to be carried out from February to March 2021. Based on this, additional feedback from all interested parties is now requested to improve the project outcomes.

Defining the scope

Figure 1: The architecture of X-Road

A critical first stage in determining environmental impacts is defining the boundary beyond which further emission sources shall not be included. In the case of X-Road, this is based on a rigorous understanding of the relevant operational and infrastructural components involved in the service, as defined in purple in the figure above. The essential operation is exchanging messages and data between two X-Road members over a pair of Security Servers. It is thought that the impact of the Central Server is likely to be quite minor and shall not be considered within the calculation.

In this context, the potential sources of emissions associated with X-Road have therefore been mapped, and can broadly be associated with the following categories:

• Life-cycle of the components used by infrastructures that allow X-Road's operations

  • Infrastructure for running a Security Server that can be on-premises or in the cloud.

  • Physical device that might be necessary for holding the keys and certificates that assure the identification of the members.

• Energy consumption of the Security Server infrastructure used by X-Road members or governing authorities.

• Energy consumption, in addition, for exchanging messages / using X-Road services.

However, a decision was reached not to include the life-cycle of different components within the footprint calculation. This is because similar studies have indicated that these areas provide only a small contribution to the total emissions relative to the use phase. This notwithstanding, a detailed analysis of X-Road's infrastructure will lead to a series of effective recommendations to minimise the life-cycle emissions. This means the calculation of emissions consists of two aspects: the operational emissions from hardware and those associated with the transmission of data.

The quantification and subsequent recommendations shall further proceed via a relative comparison between different 'use cases'. These provide a range of current and prospective parameters by which service can be defined, and thereby also a complete range of potential emission impacts within the study. Building on the experiences of Estonia and Finland, the project plans to showcase the changes to carbon footprint when changes in the following parameters are considered:

• Infrastructure level

  • The server used: On-premise infrastructure vs on cloud

  • Trust level: with and without a physical signing device

• Service level:

  • Message size across different services

  • Message log requirements (whole message, metadata only, or no message logging).
    

Call for interested parties

There have only been a limited number of studies to assess the climate impacts of specific software. The continued growth in digital services and infrastructure also means that the importance of such emissions is only likely to increase further in the future.

Beyond showcasing the proposed methodology, the project partners are further open to making use of all relevant expertise to maximise the project's success. This can be both in terms of changes to project scope, or suggestions for improving the calculation methodology. Interested parties should contact NIIS.

Report: study of the environmental impact of X-Road and the possibilities of reducing it

---

The author Peter Robert Walke joined SEI Tallinn in September 2020. He works in the sustainable development programme on a project related to developing quantitative tools for assessing the role of spatial planning policies at different scales on greenhouse gas emissions.

Peter’s background is in the physical sciences and he holds a PhD in chemistry from Katholieke Universiteit Leuven in Belgium, where he worked in nanoscience. Before that he obtained an MSci in Natural Sciences from University College London, and is currently also a postdoctoral researcher in the department of materials and environmental technology at Tallinn University of Technology. He hopes to now successfully apply his skills in research and data analysis to a new field.

Resisting Failure

X-Road is based on a distributed architecture, which makes it extremely resilient to failure. Data is always exchanged directly between a service provider and a service consumer without third parties or intermediaries having access to it. Each data exchange party may have one or more Security Servers, but data is always exchanged between two Security Servers in a one-to-one fashion. Therefore, the failure of a single Security Server only affects services available on the Security Server in question, and all other Security Servers and services of the ecosystem remain unaffected.

Image 1. X-Road is based on a distributed architecture.

Despite the distributed architecture, an X-Road ecosystem includes components that affect the availability of all the Security Servers. The good news is that the resiliency of the ecosystem against a failure of those components can be controlled and adjusted using various measures. What are the components, and how the X-Road ecosystem can be protected against their failures? Let’s find out!

Central Server

The Central Server is one of the critical components of the X-Road ecosystem. It contains a registry of X-Road member organisations and their Security Servers. Also, the Central Server contains the security policy of the X-Road instance that includes a list of trusted certification authorities, a list of trusted time-stamping authorities, and configuration parameters. Both the member registry and the security policy are made available to the Security Servers via HTTP protocol. This distributed set of data forms the global configuration that the Security Servers use for mediating messages sent via X-Road.

To be able to mediate messages, the Security Server must have a valid copy of the global configuration available all the time. The Security Server downloads the global configuration from the Central Server regularly and uses a local copy while processing messages. The Security Server remains operational as long as it has a valid copy of the global configuration available locally. It means that the Central Server may be unavailable for a limited time without causing any downtime to the ecosystem. However, registering new members or subsystems is not possible without the Central Server.

By default, the Security Server refreshes the global configuration every 60 seconds, and the configuration is valid for 10 minutes. It means that the Central Server may be unavailable for 9 minutes without affecting the ecosystem. Once the local copies of the global configuration on Security Servers expire, the message processing stops. When the Central Server starts to publish the global configuration again, the message processing continues. However, the Security Server does not queue messages or provide support for resending failed messages. It’s a service consumer’s responsibility to resend any failed messages regardless of the reason for the failure.

The global configuration download interval is configured using the “configuration-client.update-interval” parameter on the Security Server, and the default value can be overridden locally by the Security Server administrator. Instead, the global configuration validity period is configured using the “confExpireIntervalSeconds” parameter on the Central Server by the X-Road operator, and it cannot be changed on the Security Server. Therefore, all the Security Servers that are registered to the same X-Road ecosystem respect the same global configuration validity period. The download interval and global configuration validity period should be configured according to the requirements of the X-Road ecosystem. However, it is highly recommended to increase the global configuration validity period from minutes to hours or days.

Besides, the Central Server supports high availability through clustering. A Central Server cluster consists of two or more Central Server nodes. In case one of the nodes fails, the Security Server can failover to other available nodes. In a clustered environment, only a simultaneous problem with all the Central Server nodes would cause a situation where there isn’t a valid version of the global configuration available.

OCSP responder service

A certification authority (CA) issues certificates to Security Servers (authentication certificates) and X-Road member organizations (signing certificates). Authentication certificates are used for securing the connection between two Security Servers. Signing certificates are used for digitally signing the messages sent by X-Road members. Only certificates issued by trusted certification authorities that are defined on the Central Server by the X-Road operator can be used. The information about trusted certification authorities is distributed to the Security Servers in the global configuration.

The Security Server checks the validity of the signing and authentication certificates via the Online Certificate Status Protocol (OCSP, RFC 6960). An OCSP responder service providing the status information is maintained by the certificate authority that issued the certificates. Each Security Server is responsible for querying the validity information of its certificates and then sharing the information with other Security Servers as a part of the message exchange process. Only Security Servers with valid authentication certificates and members with valid signing certificates can exchange messages. If the validity information is not available or a certificate is not valid, the message exchange fails. 

To be able to mediate messages, the Security Server must have valid copies of authentication and sign certificates’ OCSP responses all the time. The Security Server downloads the OCSP responses from the OCSP responder service regularly and uses the local copies while processing messages. The Security Server remains operational as long as it has valid copies of the OCSP responses available locally, and the certificates are valid. This means that the OCSP responder service may be unavailable for a limited time without causing any downtime to the ecosystem. The period that the OCSP responder may be unavailable without affecting the ecosystem depends on various factors.

The Security Server fetches new OCSP responses using a fixed interval that is 20 minutes by default. The fetch interval is configured on the Central Server using the “ocspFetchInterval” configuration parameter by the X-Road operator. An OCSP response is considered expired by the Security Server if it was issued too far in the past OR there’s already new status information available. The validity period is defined on the Central Server using the “ocspFreshnessSeconds” configuration parameter by the X-Road operator. By default, the Security Server considers an OCSP response expired if there’s new status information available – meaning that the “nextUpdate” attribute in the OCSP response is in the past. However, the “nextUpdate” attribute can be omitted so that “ocspFreshnessSeconds” alone defines the validity period of an OCSP response. Omitting the “nextUpdate” attribute is done on the Central Server using the “verifyNextUpdate” configuration parameter by the X-Road operator.

All in all, an X-Road ecosystem’s resiliency to failures of an OCSP responder service is controlled through three configuration parameters that are all set on the Central Server by the X-Road operator and distributed to the Security Servers in the global configuration. The “ocspFetchInterval” parameter defines how often the OCSP responses are refreshed, and the “ocspFreshnessSeconds” parameter specifies the validity period of the responses, and the “verifyNextUpdate” parameter defines whether the “nextUpdate” attribute in the OCSP response is omitted. The most resilient configuration can be achieved by keeping the fetch interval short, the validity period long, and ignoring the “nextUpdate” attribute. Besides, when the “nextUpdate” attribute is omitted, it’s also possible to increase the validity period during a service break of the OCSP responder service, which buys more time to solve the problem without affecting the ecosystem. 

Besides, after the first failed OCSP request, the Security Server switches from the regular OCSP fetching interval to a failure mode during which fetching OCSP responses is attempted once a minute, by default. After the first successful OCSP request, the Security Server switches back to the regular interval.

It’s also good to be aware that not all OCSP responder services include the “nextUpdate” attribute in their OCSP responses. Usually, OCSP responder services that are based on a certification revocation list (CRL) include the attribute, but real-time OCSP services don’t. A CRL based OCSP service reads certificate statuses from a static CRL that’s refreshed regularly. In contrast, a real-time OCSP service checks certificate statuses in real-time. In case the “nextUpdate” attribute is missing from the OCSP response, the “ocspFreshnessSeconds” parameter alone defines the validity period for the response just like when the “nextUpdate” attribute is omitted using the “verifyNextUpdate” parameter.

When considering the values for the three parameters, it’s essential to consider how the values affect the evidential value of the logged messages. Since the OCSP response of the signing certificate is used to check the validity of the certificate that’s used to sign messages, the age of the OCSP response may affect the validity of the signature. Therefore, it is vital to understand the consequences that enabling the use of old OCSP responses may legally have. From a technical perspective, it is equally important that the values of the three configuration parameters are aligned with each other and the policies of the certificate authority. For example, the “ocspFetchInterval” parameter must be smaller than the “ocspFreshnessSeconds” parameter, or otherwise, the Security Server considers the responses expired before new ones are fetched.

Time-stamping service 

All the messages sent via X-Road are time-stamped and logged by the Security Server. The purpose of the time-stamping is to certify the existence of data items at a certain point in time. A time-stamping authority (TSA) provides a time-stamping service that the Security Server uses to time-stamp all the incoming/outgoing requests/responses. Only trusted TSAs that are defined on the Central Server by the X-Road operator can be used. The information about trusted TSAs is distributed to the Security Servers in the global configuration. The approved time-stamping authorities must implement the time-stamping protocol (RFC 3161) supported by X-Road.

By default, X-Road uses batch time-stamping, which means that new messages that have been processed since the previous batch time-stamping and do not have a time-stamp yet, are time-stamped once a minute. The time-stamping interval is defined on the Central Server using the “timeStampingIntervalSeconds” parameter by the X-Road operator, and it cannot be changed on the Security Server. If the time-stamping fails, the Security Server continues to process messages until the acceptable time-stamping failure limit is reached. By default, the limit is 4 hours, and it’s configured on the Security Server using the “message-log.acceptable-timestamp-failure-period” parameter. The default value can be overridden locally by the Security Server administrator. When the limit is reached, the Security Server quits processing messages. When the time-stamping service becomes available again, all the messages missing a time-stamp are time-stamped, and the Security Server continues normal operations.

Besides, after the first failed time-stamping attempt, the Security Server switches from the regular time-stamping interval to a failure mode during which time-stamping is attempted once a minute, by default. After the first successful time-stamp, the Security Server switches back to the regular time-stamping interval.

The Security Server also supports automatic failover between time-stamping services if it has more than one configured time-stamping service. It means that the Security Server tries time-stamping with all the configured services until time-stamping succeeds or all the configured services have failed. The behavior is repeated for every batch.

Alternatively, the Security Server can be configured to time-stamp messages synchronously. It means that every message is time-stamped immediately, and if time-stamping the message fails, processing the message fails too. In case a security policy requires that every processed message is time-stamped within a defined time window, this configuration option can be used to guarantee it. However, the downside of synchronous time-stamping is that it increases the load of the time-stamping service tremendously compared to batch time-stamping. When the batch time-stamping is used, the load does not depend on the number of messages exchanged over the X-Road. Instead, it depends on the number of Security Servers in the system. Another downside of the synchronous time-stamping is that it increases the processing time of each message since the time-stamping is done synchronously as a part of the message processing flow. It means that four time-stamping operations are added to the end to end processing time of each message.

All in all, an X-Road ecosystem’s resiliency to failures of a time-stamping service is managed through different factors. The time-stamping interval and number of available time-stamping services are defined on the Central Server by the X-Road operator. Instead, the acceptable time-stamping failure period and the time-stamping mode (batch / synchronous) are defined on the Security Server, and the default values can be overridden locally by the Security Server administrator. The most resilient configuration can be achieved by using batch time-stamping, keeping the time-stamping interval short, keeping the acceptable time-stamping failure period long, and configuring multiple time-stamping services on the Security Server.

However, just like with the OCSP related configuration, it’s essential to consider how the selected values affect the evidential value of the logged messages. For example, the age of a time-stamp may affect its evidential value from a legal perspective. Also, whether it is acceptable to have messages without a valid time-stamp must be considered, and the time-stamping mode (batch / synchronous) should be selected accordingly.

Conclusions

An X-Road ecosystem is exceptionally resilient to failure. Different components may fail separately or at the same time, and the ecosystem is still capable of processing messages and transferring data. How long a single component may be unavailable without affecting the ecosystem depends on the configuration of the ecosystem and the configuration of individual Security Servers. The X-Road operator is responsible for defining and managing the ecosystem’s configuration. Still, the Security Server administrators may define some configuration items locally since the requirements may vary between organisations and Security Servers.

The values of different configuration items vary between X-Road ecosystems, and they depend on the requirements and constraints regarding availability, the evidential value of the logs, costs, etc. Also, financial factors play a role when defining the OCSP fetch interval and time-stamping interval since some commercial trust service providers request a transaction-based fee for the use of their services. In those cases, costs can be optimized by adjusting the intervals without forgetting the legal requirements regarding the age of the OCSP responses and time-stamps. All in all, the configuration should be in balance between different requirements and constraints. Sometimes it may require compromises between objectives.

Security Server Sidecar (part 3)

This is a series of blog posts about X-Road® and containers. The first part provides an introduction to containers and container technologies in general. The second part concentrates on the challenges in containerizing the Security Server. The Security Server Sidecar – a containerized version of the Security Server – is discussed in the third part.

Security Server Sidecar is a containerized version of the Security Server that supports production use. The Sidecar is a Docker container that runs in the same virtual context (virtual host, Kubernetes Pod, etc.) with an information system. The containerized approach makes running the Security Server more cost-effective since no separate host server needs to be allocated for each Security Server. Besides, more and more information systems exchanging data using X-Road are running in containers too, so it’s beneficial to be able to run the Security Server on the same platform with the information systems that are connected to it.

The Sidecar solves the challenges related to running the Security Server in a container, and it uses the standard release versions of the Security Server software. In other words, the Sidecar is built from pre-built packages of the official X-Road releases, and it is a separate project that builds on the X-Road core.

The Sidecar project

From an administrative perspective, Security Server Sidecar is a project of the Finnish Digital Agency (DVV) that is implemented in collaboration with NIIS. The DVV owns the project, and NIIS is responsible for coordinating the daily development activities. All the deliverables are released on NIIS’s GitHub and Docker Hub accounts. The project is currently ongoing, and it will be completed by the end of 2020.

The project will produce a Security Server Sidecar Docker image with a couple of alternative configurations. The Sidecar slim is a lightweight version of the Security Server, and it does not include message log, operational monitoring, and environmental monitoring modules. It means that the slim version does not log messages or provide any monitoring capabilities. However, technically it can be used for both consuming and providing services if the capabilities mentioned before are not required.

Instead, the regular Sidecar includes message log, operational monitoring, and environmental monitoring modules too. Similarly, the regular Sidecar can be used for both consuming and producing services. The Sidecar slim is a lightweight version of the Security Server while the regular Sidecar provides all the features of a full-blown Security Server installation. Besides, also versions with country-specific meta-packages are available. Currently, the only country-specific configuration available is the Finnish meta-package.

In addition to the Security Server Sidecar Docker image, the project also produces documentation to support the use of the image. The documentation will cover best practices and examples of how to run the image on a Kubernetes cluster using Elastic Kubernetes Service (EKS) on the Amazon Web Services (AWS) cloud platform.

What is a sidecar? 

In general, the sidecar is a design pattern commonly used in a microservices architecture. A sidecar is an additional component that is attached to a parent application to extend its functionalities. The pattern aims to divide the functionalities of an application into separate processes. The approach allows adding new capabilities to an application without changing the application itself. In this way, a sidecar is loosely coupled with the application. For example, logging and monitoring are functionalities that are often implemented using a sidecar.

Applying the sidecar pattern to the Security Server

When using the regular Security Server version on a Linux host, it’s strongly recommended that the Security Server is running on its own host and not on the same host with an information system that is connected to it. It means that at least two separate hosts are required. Instead, the idea of the sidecar architecture pattern is that an application and a sidecar run on the same host or context, close to each other. With the containerized Security Server, the goal can be achieved since the Security Server is packaged in a container that runs in its own isolated process.

Image 1. The Security Server Sidecar and an application in the same virtual context.

The original idea of the sidecar pattern is that multiple copies of the same sidecar are attached to the application so that each instance of the application has its own sidecar. In case different applications use the same sidecar, the same approach applies to all the applications and their instances.

Image 2. A single Security Server Sidecar instance is shared between multiple instances of an application, and between different applications.

Despite its name, the original sidecar pattern does not work very well with the Security Server Sidecar since the Sidecar requires the same configuration and registration process as the regular Security Server. Also, even if the Security Server is containerized, the footprint of the Sidecar container is still relatively massive compared to the footprint of average containers. Therefore, it’s recommended that a single Sidecar container is shared between multiple instances of the application, and it may also be shared between different applications too. For high availability and scalability, a Sidecar cluster consisting of a primary node and multiple secondary nodes can be considered. Let’s take a better look at the different deployment alternatives next.

Running the Sidecar on Kubernetes

The Sidecar can be deployed to different container management systems, thanks to standardization. One of the most popular container management systems is Kubernetes, which is available as a service on multiple cloud platforms, such as Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform. Kubernetes is open-source, which means that it can be used in on-premise and private cloud environments too. In this case, we’re going to concentrate on running the Sidecar on AWS Elastic Kubernetes Service (EKS). During the development project, the Sidecar has been tested using the Docker Engine and EKS. 

Since the Security Server is a stateful application, it is required that an external database and persistent file storage are used in all the deployment alternatives. In this case, Amazon Relational Database Service (RDS) is used for the Security Server databases, and a Kubernetes persistent volume is used to store the configuration files.

Before going into the deployment models, a few words about Kubernetes and Pods since Pods play an essential role in the deployment models. In Kubernetes, a Pod is a group of one or more containers that run in a shared context, share the same storage and network resources, and share a specification of how the containers are run. Each Pod runs a single instance of an application, and scaling the application horizontally means using multiple Pods, one for each instance of the application.

Deployment is another Kubernetes concept that’s essential to the Sidecar deployment models. A Kubernetes deployment represents an application that consists of a set of identical Pods. The deployment specification defines the configuration of the Pods and the number of replicas to run. The deployment maintains the Pods and monitors that there’s a correct number of Pods running. Also, it’s possible to create a horizontal autoscaler for a deployment that automatically scales the number of running Pods based on the selected metrics.

Security Server as a sidecar

Image 3. The Security Server Sidecar as a real sidecar inside the same Pod with an information system.

The first alternative is to deploy the Sidecar as a real sidecar, which means deploying it in the same Pod with an information system. It is a feasible approach if there’s always only one Pod running, and other information systems do not need to access the Security Server. The information system can be a service consumer, service producer, or both.

In case the information system must be scaled horizontally, this approach does not work very well. The reason is that adding a new Pod means that the new Security Server running in the Pod must always be configured and registered before it can be used. Since the onboarding process of the Security Server may take from days to weeks, the approach is not feasible. Also, deploying a Security Server for each Pod would generate quite much overhead from the resource consumption perspective.

Single Security Server

Image 4. The Security Server Sidecar in its own Pod and shared by multiple information systems.

When multiple information systems or several instances of the same information system need to access the Security Server, it’s better to deploy the Security Server in a separate deployment using a single Pod. In this way, the information systems can be scaled independently from the Security Server, and some of them might even be running outside of the AWS EKS cluster. However, the number of Security Server instances is limited to one. Since the external database and persistent volume are used, Kubernetes can automatically recover the Security Server Pod in case of failures. Also, in this case, the information systems can be service consumers, service producers, or both.

In case multiple Security Servers are required because of high availability and/or scalability, there are two alternatives: multiple independent Security Servers or a Security Server cluster. Multiple independent Security Servers mean deploying several Security Servers, with each of them having a unique identity. This approach provides high availability. Instead, a Security Server cluster means deploying a group of Security Servers that share the same identity and that are accessed through an external load balancer. The cluster provides both high availability and scalability. More information about X-Road’s load balancing alternatives can be found here.

Multiple Security Servers

Image 5. Multiple instances of the Security Server Sidecar shared by several information systems.

Deploying multiple independent Security Servers provides high availability but not scalability from a performance point of view. In this setup, multiple Security Servers with unique identities are deployed as separate, independent applications. In practice, Security Servers are deployed using separate deployments, which means that they have their own run specifications. Besides, the number of the Security Server Pods within a deployment is limited to one for each Security Server. Adding a new Security Server to the setup means creating a new deployment plus configuring and registering the newly created Security Server.

The information systems can be service consumers, service producers, or both. Service consumers may connect to the Security Servers directly, or there may be another load balancer between the consumer information systems and the Security Servers (omitted in the diagram). In the case of service producers, Security Server’s internal load balancing enables publishing services on multiple Security Servers and routing service requests to all of them. However, the configuration (e.g., available services, access rights) must be manually synchronized between Security Servers providing the service. Only the Security Server cluster provides automatic synchronization between Security Servers.

Security Server cluster

Image 6. Security Server Sidecar cluster with an external load balancer shared by multiple information systems.

A Security Server cluster provides both high availability and scalability. It consists of a primary node and one or more secondary nodes that all share the same configuration and identity. In this setup, the primary node is used to manage the cluster, and it does not process messages. In practice, configuration changes are done on the primary node, and they’re automatically replicated to the secondary nodes. Replication covers the configuration database and configuration files. Changing the configuration on the secondary nodes is blocked. The secondary nodes are connected to a load balancer that distributes incoming traffic between them. Further implementation details of the Security Server cluster on Kubernetes are studied in more detail in the Sidecar project.

The information systems can be service consumers, service producers, or both. Service consumers may connect to the secondary nodes directly, or there may be another load balancer between the consumer information systems and the secondary nodes (omitted in the diagram).

Multiple Security Servers or a Security Server cluster?

The key difference between multiple independent Security Servers and a Security Server cluster is that the cluster provides both high availability and scalability. In contrast, independent Security Servers provide only high availability. In the cluster, secondary nodes can be scaled with less effort while setting up a new independent Security Server is a manual operation. Also, in the cluster setup, all Security Servers share the same identity and configuration that is synchronized automatically. Instead, multiple independent Security Servers each have their own unique identity and configuration, and there’s no synchronization between them. Which one is the best alternative depends on the use case and its requirements.

Containerized future?

The four deployment models described before give an overview of what kind of models can be considered for the Sidecar. The same models can be applied regardless of the underlying platform or environment where the Security Server is deployed. Of course, the implementation details vary between different platforms and environments, but the high-level architecture patterns remain the same. However, the models do not provide an exhaustive list of available alternatives since different models can be combined and new elements, such as load balancers, can be added to the described ones.

Adding support for containers does not mean dropping support for Linux – running the Security Server on Ubuntu and Red Hat will remain supported in the future too. Containers are an alternative way to run the Security Server, and X-Road members are free to choose between the available alternatives. Containers are a convenient way to run the Security Server when an organization already has the required capabilities to operate and manage containers in production environments. However, in case an organization is not quite there yet, using virtual machines might be a better alternative since mastering Security Server containers on a production level requires time, effort, and experience.

It must also be noted that the Security Server configuration process – registering and onboarding a fresh Security Server to an X-Road ecosystem – is always the same regardless of the Security Server packaging. From a process perspective, the containerized version of the Security Server is not different from the Linux packaged version. In this way, X-Road members can be sure that the same level of trust is always guaranteed in data exchange between X-Road members.