Technical Articles



International Journal of Auditing – Julyl 2007

Planning and Selecting Performance Audits at the Netherlands Court of Audit

All performance auditors share the same goal: to audit the most relevant problems using the right people and the most appropriate methods and techniques at the right moment and to use the audit findings to make a meaningful contribution to the quality of government policy and operational management.

The Netherlands Court of Audit devotes a great deal of time to the proper planning and selection of its audits. We work in a very broad field but with finite capacity. The Court of Audit’s mandate covers all of the central government and nongovernmental institutions that receive public funds to carry out statutory tasks. In all, we must audit the use of about 350 billion euros with a staff of just 80 auditors who specialize in performance audits. For staffing reasons alone, we have to think very carefully about which performance audits we will carry out. More importantly, if our work is to be effective, we have to know where we can make the greatest contribution.

In this article, we explain how the Court of Audit tries to realize its goals. The first section describes how we use the planning and selection tools in practice—which tools we use, when we use them, and how they are related to each other—and the next section includes the results and the lessons learned from our use of the planning and selection tools.

Planning and Selection Tools

At our SAI, audit planning and selection must help us achieve the Court of Audit’s mission: to “audit and improve the regularity, efficiency, effectiveness, and integrity with which the State and associated bodies operate.” Another task of the Court of Audit is to “contribute to sound public administration through cooperation and knowledge exchange at home and abroad.” For purposes of this article, it is also relevant that we seek to be a transparent organization.

To plan and select our performance audits, we use a number of tools that are closely related to the Court of Audit’s overall strategy (see figure 1).

Figure 1: Netherlands Court of Audit Strategy and Audit Planning and Selection Tools

Figure 1: Netherlands Court of Audit Strategy and Audit Planning and Selection Tools

Planning and selection begins with the Court of Audit’s strategy. Three domains— public services, safety and security, and sustainable development—have been prioritized to implement the strategy, and they determine the most relevant objects for audit. We also use three tools—monitoring, issue matrix, and integrated risk analysis—to look beyond the strategy at more topical developments in our field of operations.

The first tool is monitoring, a standard activity carried out by all Court of Audit organizational
units. In all events, we monitor the ministries’ policies and the performance of officials charged with statutory tasks. Given our strategic goal of helping to reduce social problems, we also increasingly monitor social developments.

The issue matrix, the second tool, is an outcome of monitoring and is intentionally designed to facilitate discussion of the monitoring findings with the Court of Audit’s Board. Issues are summarized in memos that answer the following four questions: (1) What is at issue? (2) Is it undesirable? (3) Where does it occur? (4) Who are the main players? The issue matrix identifies relevant issues in our audit field that are not necessarily covered by our strategy.

Figure 2 is an example of an issue matrix in the transport, public works, and water management policy field. The horizontal axis shows the likelihood of an event’s occurrence,
and the vertical axis shows the potential impact of an event. By ordering the issues in such a matrix, we are better able to determine their potential importance. To encourage further discussion, we deliberately do not define the concepts of “potential impact” and “likelihood of occurrence” any further.

Figure 2. Example of an Issue Matrix for Transport, Public Works, and Water Management Policy

Figure 2: Example of an Issue Matrix for Transport, Public Works, and Water Management Policy


According to this issue matrix, it is our opinion that accessibility (motorway congestion) is almost certain to occur and will probably have a high impact. In the Netherlands, it is indeed a daily problem: overcrowded rush-hour motorways harm both the environment and the economy. Flooding, which may also have a high impact, is thought to be less of an issue because it is less likely to occur. The privatization of the national airport, Schiphol, would have a low impact since it is not expected to seriously affect the continuity of the airport’s operations. We have not identified issues with a low likelihood of occurrence in this policy field, though they may exist, Rather, the Court of Audit is more interested in issues that are more likely to go wrong.

The third tool is integrated risk analysis. In contrast to monitoring and issue matrices, risk analysis builds on the strategy and is more focused so that it can be used directly for the activity program and planning of regularity audits.

Integrated risk analysis (IRA) is a systematic and efficient means to generate, analyze, and record information about the entire audit field to identify and classify risks in the domains that are relevant to the Court of Audit. The risk analyses are integrated in that they

  • combine all identified and prioritized risks so that choices can be made for both
    the regularity audits and the annual programming of performance audits and
  • are conducive to detecting connections between public administration operations
    and risks to public administration performance.

The system used to calculate risk is very similar to the one used for issue matrices. The risk is estimated as a combination of the likelihood of an undesirable event or situation occurring and its impact on one of the risk domains: risk = {likelihood of risk x impact of risk}. We use the model shown in figure 3.

Figure 3: Integrated Risk Analysis Matrix

Figure: 3 Integrated Risk Analysis Matrix

The end product of integrated risk analysis is a systematic overview of the main risks to the operation and performance of public administration and associated third parties. The analysis results serve as input in our proposals for regularity audits (selecting the auditee and approach) and the annual programming of performance audits. Integrated risk analysis also contributes to accumulating field-specific know-how and exchanging that know-how among organizational units.

Risks to public administration operations and performance may be related to each other. For example, IT problems (cause) at the Ministry of Finance may lead to late payments of housing benefits (effect). This might indirectly lead to another ministry’s failure to achieve a policy goal, such as reducing the number of forced evictions. A simplified cause and effect diagram for this example is given in figure 4.

Figure 4. Example of Potential Cause and Effect of Risks to Public Administration Operations and Performance

Figure 4: Example of Potential Cause and Effect of Risks to Publix Administration Operations and Performance

The tools we use to plan and select performance audits ultimately generate input for programming. Ideally, by comparing the outcomes of monitoring and issue matrices (which have a broader and more topical orientation) with the proposals arising from the strategy (which have a more focused and longer term orientation), we will program the right performance audits at the right moment. The direct result of our planning and selection activities is therefore an optimal activity program, which in turn determines how effective we are as an audit institution. However, that is sometimes beyond our control.

Conclusions

This article has explained how the Court of Audit is trying to perfect its planning and selection of performance audits. We are pleased with the three benefits we have identified in the planning and selection process. First, the process produces a systematic and lively internal and external debate. Second—and this is a significant advantage of our multiyear strategy—it enhances our external profile. An auditee sometimes sighs, “Yes, if I read your strategy and hear your arguments, it goes without saying that you’ll want to audit this.” And although we prefer not to hear people sigh, this often increases their cooperation and willingness to take a critical look at their own policy practices. Third, it encourages rigorous, activity-based management, both in the three domains recognized in our strategy and in auditing the relationship between policy and its implementation, the main focus of our performance audit strategy.

People sometimes see an overlap in our preparations: in addition to carrying out monitoring activities and integrated risk analyses, we have identified special domains and must prepare related proposals. We, too, sometimes think that we have set our goals too high: we must invest a considerable amount of time in the planning and selection process, yet the capacity available to carry out further audits is limited. We also have to respond to current evelopments and requests from Parliament. Finally, programming our audits in response to social problems is a daunting challenge. Sometimes, operational audit “magnets” divert us from finding the causes of disappointing results and lead us to check procedural and organizational systems. Fortunately, we are getting better at resisting these modern-day “Sirens.”

For additional information, contact the authors at r.turksema@rekenkamer.nl and
peter.vanderknaap@rekenkamer.nl.

References

Netherlands Court of Audit (2003), Performance and Operation of Public Administration: Strategy 2004-2009 of the Netherlands Court of Audit, The Hague, Netherlands Court of Audit.

de Vries, G.J.D. (2000), Beleidsdynamica als sociale constructie. Een onderzoek naar
doorwerking van beleidsevaluatie en beleidsadvisering
, Delft, Eburon.