The System-Oriented Approach to Performance Auditing: An Underrated Option

by Jan van Dam, Netherlands Court of Audit

Introduction

Pursuant to International Standards of Supreme Audit Institutions (ISSAI) 300-26, performance auditors generally choose between a result-, problem-, or system-oriented approach (or a combination thereof) to facilitate the soundness of audit design.

In this article, the Netherlands Court of Audit (NCA) chronicles its performance audit experience according to the system-oriented approach (dating back to the 1980s) and what Supreme Audit Institutions (SAIs) can learn from that experience today, especially SAIs that want to build performance audit capacity but don’t know where to begin.¹

Why a System-Oriented Approach?

Performance audits, conforming to a system-oriented approach, focus on the proper functioning of management systems. In supporting performance audit capacity development in many countries, the NCA has seen that a system-oriented approach helps to focus the audit on structural issues, not on incidents; on the big picture, not on details; and on the performance of government and public organizations, not on legislation.

The NCA has found that financial and compliance auditors tend to think in terms of rules and regulations, and that the required change in mind-set to conduct performance audits is quite a big step and needs time.

A system-oriented approach can be of assistance in bridging both disciplines and provides a potentially more impactful way (value and benefit) as a SAI. The focus is on real, structural problems; it delivers more relevant audit results to the auditee; and it generates SAI visibility in the public sector and society.

The system-oriented approach does not focus primarily on the policy or the goals but on well-functioning management systems as a condition for effective and efficient policies. Examples are financial management, evaluation, control and Information Communications Technology (ICT) systems.

This type of audit can use descriptive questions (which also help to detect risk areas in the system), such as:

• What is the objective of the system?
• Who are the responsible actors within the system?
• What are the responsibilities of each actor?
• Which rules, regulations and procedures are relevant?
• What are the relevant money and information flows?

…and can be complemented by more evaluative questions, to include:

• To what extent is there a sound plan?
• Is there a good quality monitoring system?
• Is the comparison between the monitoring system information and the plan leading to adjustments, if necessary?
• Is planning, monitoring and adjusting recorded in a systematic way, ensuring accountability to a higher administrative level?
• Are processes evaluated periodically in a proper way?

A Step-By-Step Approach. Where to Begin?

When an audit subject has been chosen based on sound risk analysis, describing the system under audit is a good start and may also result in an interesting report of facts.

For example, in the Netherlands, programs and money flows are sometimes so complicated that a clear description of the system (preferably in schemes and chart flows) is helpful to MPs and even auditees. This is referred to as a clarification audit, and these have been very successful. Here, the audit questions have a descriptive character.

Figure 1 represents a chart flow example. It visualizes the expenses for the Netherlands railway system, as well as the responsible actors. The system complexity is evident.

An important and useful first audit step in a system-oriented approach—when it comes to a normative audit—is to ask a question related to: “What conditions must be met to ensure the economy, efficiency and effectiveness of the system?”

This conditional approach can be applied to a variety of audit subjects, such as state-owned enterprises, social security programs, a tax system and a public school. In this case, the central audit question is, “To what extent does the auditee fulfill the conditions for economy, efficiency and effectiveness?” It is interesting to identify these conditions and how these conditions can be structured to make them comprehensible to the early-stage performance auditor.

Based on NCA experience, looking from the system perspective is very helpful when defining and structuring these conditions. One of the first times the NCA chose to embark on a system-oriented approach was a government-wide performance audit on subsidies management (published in 1989). The provision of subsidies was perceived as a system (see Figure 2).

This approach is further developed and generalized in Figure 3. The auditee is positioned at the top. This is very important, as performance auditors must view the situation from the auditee perspective. What must the auditee do to be efficient and effective?

Here, two main activities are distinguished:

• Acting/steering, which equates to moving in the right direction; and
• Gathering and receiving relevant, reliable and timely information.

Acting and gathering information can be aimed at a variety of subjects, either organizations (such as state-owned enterprises, a hospital or a school) or processes (such as a housing policy, social security program or tax system).

Figure 4 illustrates the organizational set-up. Starting from the bottom, the diagram outlines an entity with inputs, throughput, output and outcome.

Using a road construction company as an example (acting as a state-owned enterprise), the input would consist of raw materials (sand, cement, asphalt, but also labor). The throughput (process) is construction (of a road, which includes leveling, paving and blacktopping). The output is a road in good condition, and the outcome could be the opening up of a region that was previously difficult to reach.

In keeping with the approach as outlined in Figure 4, the minister (or the agency responsible) should act—setting goals and criteria for the road construction company—and should be informed by enterprise management about activities and results. Further, the minister or agency must take corrective actions if the information suggests cause for concern.

The system approach can also be used for processes (programs or policies), as shown Figure 5. In this instance, a well-known policy or program cycle is used. Consider, for example, a social security program. The minister or agency must prepare this program carefully by:

• Evaluating pros and cons of different instruments and options.
• Setting smart goals and clear criteria.
• Establishing the budget according to stated ambitions.
• During implementation, the legitimacy and effectiveness of benefits should be guarded, as well as the efficiency of the process.
• The last step is to evaluate the economy, efficiency and effectiveness of the social security program itself, including the follow-up of recommendations to improve the program. This completes the cycle.

Conclusion

SAIs are in a position, facilitated by the International Organization of Supreme Audit Institutions (INTOSAI) community, to genuinely benefit from mutual experience through sharing lessons learned on implementing performance auditing. For inexperienced performance auditors, economy, efficiency and effectiveness are quite abstract concepts. Building performance audit capacity, therefore, is a challenge. However; based on the experience of the Netherlands Court of Audit (NCA), a system-oriented approach seems to be a practical first step into the challenging world of performance audit.

References

ISSAI 3100 Appendix (Building a Performance Audit Function)