Testing plays an important role in every software project and its aim is to verify that the software meets its requirements and find errors and other defects in the software. X-Road is no exception – the data exchange layer is a critical component of e-government architecture and it is used nationwide in multiple countries so there’s no room for software errors.
In a traditional waterfall software development project, testing is typically done during a separate testing phase once requirements have been defined and the code implementing them has been programmed. In agile software development projects testing is usually done concurrently with requirements and programming. The benefit of the agile approach is that problems are detected faster which makes fixing them easier and therefore also cheaper. X-Road core is nowadays developed using agile development methods and testing follows the same approach. The aim of X-Road core testing is to automate everything that can be automated and catch all the errors as early as possible. Automation helps to reduce manual work, but it also ensures that processes are repeated in the exact same way each time.
Background
X-Road has around 50 000 lines of code so manual testing is really not an option. The Finnish Population Register Centre and Estonian Information System Authority, that jointly developed X-Road before the development was handed over to NIIS in June 2018, invested a lot of effort in automated testing and code quality in the recent years. For example, unit test coverage increased from 29 % (2015) to 40 % (2018). At the same time technical debt ratio decreased from 5.3 % to 2.6 %. Total test coverage also increased significantly, but unfortunately no exact numbers are available as measuring it is still a work in progress.
X-Road core testing now
The tests were handed over to NIIS together with X-Road core in June 2018. NIIS is responsible for testing the X-Road core that includes Central Server, Configuration Proxy, Security Server and external APIs and interfaces provided by the core components. Organizations operating their X-Road environments are responsible for testing any specific hardware and their own extensions built on top of the X-Road, e.g.: API catalog, environmental monitoring tools, operational monitoring tools etc.
Image 1. X-Road development and testing tools
Just like the software requirements, test cases and test results are managed in JIRA issue management system. Most of the tests are automated and there are three alternative execution schedules:
when new code is submitted
daily
before releasing a new version.
The execution schedule of a test depends on the type and nature of the test. For example, unit tests are executed every time when new code is submitted, but performance tests are executed daily. There is no sense to execute all the tests on every change only because it can be done.
Automated tests are divided in the following categories:
unit testing
API / interface testing
UI testing
performance testing
security testing
source code quality control and static analysis
installation and upgrade testing.
New releases must pass all the categories and tests before they are published and distributed to NIIS members and X-Road community.
Image 2. X-Road development, testing and distribution
In addition to automated testing, some areas require manual work:
internal code reviews – developers review each other’s work (ongoing)
security audits – source code + white hat hacking (periodical)
licence compliance checks (periodical).
X-Road core development and testing is done on Amazon cloud which enables optimizing resource usage and maximizing automation. In practice, this means that all the required resources can be created and destroyed automatically, and resources need to exist only when they are actually being used. For example, there’s no need to clean up a performance test environment between test runs as a brand new environment can be automatically created for each run.
The road ahead
X-Road is based on distributed architecture and many use cases contain complex processes that require operating multiple components asynchronously, e.g. registration of a new Security Server requires operating Central Server, Security Server and Certification Authority. Despite the complexity most of the use cases have automated tests that can be executed without manual work as often as it's required. When the software changes the tests must be changed accordingly, of course. Just like software development, also testing is continuous work that is not about to end any time soon.
NIIS will continue to develop all the aspects of the quality assurance of X-Road. Currently NIIS is looking for a testing partner to provide resources for X-Road core software testing. The procurement is open until the end of July 2018 and general information about the procurement can be found in the Estonian e-procurement portal at https://riigihanked.riik.ee/register/hange/197345.