Highlights

International Journal of Government Auditing – July 2014


The IT Audit of the Republic of Slovenia’s eHealth Project

Implementations of information technology (IT) support for public services and functions are significant public investments. Consequently, implementation shortcomings and glitches in these projects usually receive high public and media attention. In 2013, several IT support implementation issues even reached the global media: in the United States, upgrades to the unemployment benefits systems in Florida, Pennsylvania, California, and Massachusetts left hundreds of thousands of unemployed citizens without unemployment benefits; also in the United States, the website in support of the Affordable Care Act did not operate efficiently for several months; and, in Australia, Queensland Health, which administers the public health system of the state of Queensland, suffered when its payroll implementation caused thousands of workers to be underpaid, overpaid, or not paid at all. While public expectations regarding IT support projects are high, even the best planned projects remain exposed to numerous risks—and are prone to all-too-frequent hiccups, if not all-out failures.

Like other countries, the Republic of Slovenia has had its fair share of public services’ IT support issues. The Court of Audit of the Republic of Slovenia is Slovenia’s highest body for the supervision of state accounts, the state budget, and all public spending, and has therefore conducted a number of IT support performance audits. The Court of Audits has focused these audits on IT effectiveness and efficiency, and its independent assessments and analysis of IT support have covered diverse public service areas such as tax and customs systems, public health, agricultural policy, and social benefits. In several cases, the Court of Audits’ findings have served as the basis for decisions on future development and activities in the public service.

eHealth project in the Republic of Slovenia

While many failed IT support implementations recently achieved widespread notoriety, one of Slovenia’s largest IT projects had long remained hidden from the public’s eye. The Ministry of Health’s eHealth project was aimed at modernizing all aspects of the public health services with a number of new, interconnected IT support solutions. The project promised to 1) connect thousands of health practitioners with a safe communication network, 2) standardize and unify the exchange of patients’ records and documentation, 3) open new communication pathways for patients and health services’ practitioners, and 4) introduce advanced health-related knowledge databases and decision support mechanisms.

The project officially began in 2005; however, project plans were not finalized until 2009. The plan was to gradually introduce new IT solutions each year, through 2015. The cost of the development, implementation, and initial support of these IT solutions was estimated at 67.5 million Euros, with an additional 65.5 million Euros budgeted for support upgrades until the end of 2023.

By the middle of 2012, the project had not reached any of its set milestones. The Court of Audit decided to carry out a performance audit of the eHealth project, to identify the reasons for the delays and the potential risk that the project would not deliver expected benefits in the budgeted timeframe.

The Court of Audit’s approach

To understand whether the Ministry had been effective in managing the eHealth project, the Court of Audit needed to develop a customized approach to offset the fact that at the time of the audit the project was not yet finished. Large IT support implementation projects are frequently managed in line with internationally recognized project management frameworks that can be used to develop a practical IT project performance audit approach. The Ministry, however, had not used any formalized standards in the planning and management of the eHealth project. The Court of Audit therefore focused on only four project management areas: 1) content of the eHealth project, 2) the project’s organization and management of human resources, 3) management of the project timeline, and 4) management of project finances.

Challenges

The content of the eHealth project had been changed several times, and the records of these changes were limited. This meant that neither the scope nor the expected results—the project’s IT solutions— were clear.

By the end of 2012 the Ministry had spent at least 8.8 million Euros on the eHealth project. Due to the project’s lack of transparency, the Court of Audit frequently could not reliably connect the Ministry’s purchase of hardware, software, and various services to specific project requirements— making it difficult to determine whether the purchases were justified.

During its long course, the eHealth project had been run by several project managers. Neither their authority nor the authority of other project stakeholders had been clearly defined. In several cases the Court of Audit had not been able to determine who authorized important content-related changes. The transparency of the project was further reduced by documentation that was inconsistently updated and disorganized, and which frequently existed in a number of incompatible versions. Obtaining reliable information on past events was therefore extremely demanding.

To bring some order into the chaos, the Court of Audit invested a significant effort into organizing the scattered and incompatible information on the state of the eHealth project into a clear and concise picture. Incomplete project documentation always presents a risk of failing to discover and disclose significant irregularities. To mitigate this risk, the Court of Audit reviewed several hundred documents in various document repositories (not necessarily connected to the eHealth project), in the hope of gathering as much information as possible. The Court of Audit also increased the samples of all tests related to payments – in some cases reviewing all payments related to a specific part of the project.

Audit results

The Court of Audit determined that between January 2004 and September 2013 the Ministry had not effectively managed the eHealth project. Although by September 2013 the Ministry had managed to connect the largest public health institutions with a communication network, it failed to introduce any of the originally planned IT solutions or develop standardized patient records. And, in total, the Ministry enabled only 10 (out of hundreds) health practitioners to digitally exchange patient documents. The Court of Audit thus concluded that none of the originally planned project milestones and deadlines had been met.

The Court of Audit also assessed that before 2013 the eHealth project had not been effectively organized. As the project utilized numerous external consultants, it was frequently not clear who the members of the project team were or what roles they were to carry out. As the authority of various project stakeholders was not clearly defined, their personal accountability was effectively reduced.

This, in turn, led to a diminished oversight over project spending. For example, the Ministry purchased hardware and software, but then, for up to two years, failed to set it up. Despite not using these tools, the Ministry paid for their maintenance and upgrades. The Ministry also paid for numerous external consultants whose contribution to the project was frequently unclear. The Ministry funded at least one software solution that had been implemented in only two public health organizations (and had then been abandoned), and paid for the development of several software solutions with overlapping functionalities. The Ministry even paid for upgrades to existing health sector software solutions, owned by commercial software companies, for which the public health services providers already pay license and maintenance fees. As the Ministry gave the supposed technical deficiency of these same solutions as one the primary reasons for starting the eHealth project in the first place, investing in their upgrades appeared inconsistent with the undertaking of the entire project.

Lessons learned

Although the levels of IT governance vary vastly from country to country, supreme audit institution auditors who plan to undertake a performance audit in the area of IT implementations might consider the following audit risks:

  • Establishing accountability for key decisions in the project might prove difficult. Why? Because the setup of any project can change significantly according to the organization of a government ministry. The project’s organization, and the authority and responsibility of the project’s participants, should be defined in the project documentation. However, the organization of a government ministry is governed by bylaws and the Ministry’s internal regulations— so these documents might be misaligned. Also, projects typically last longer than ministerial terms in office, and subsequent personnel changes might further blur the issue of accountability.
  • Due to poor management of a project’s documentation, the timeline of key events, the parties involved, the authorizations given, and even the scope of the project, could all be difficult to establish. A government ministry manages its documentation in accordance with official guidelines, which might be inappropriate for efficient project management.
  • If the project has been financed from multiple sources, calculating its complete cost might prove difficult and time consuming.
  • If numerous contractors have been involved in the project, and if their contracts have not been appropriately prepared, the definition of roles in the project is not always conclusive.
  • Terms related to IT support are often vague. Unless all the project terms are clearly and unambiguously defined, determining whether single tasks within the project have achieved their defined goals can be difficult.

Conclusion

In 2013 the Ministry has scaled down the eHealth project plans. It has improved the project’s organization, its spending oversight, and the personal accountability of the project team. It is, however not yet clear whether the project will realize its expected benefits. The Court of Audit will continue to monitor the project for further developments.