International Journal of Government Auditing – Spring 2017
Pursuant to International Standards of Supreme Audit Institutions (ISSAI) 300-26, performance auditors generally choose between a result-, problem- or system-oriented approach (or a combination thereof) to facilitate the soundness of audit design.
In this article, the Netherlands Court of Audit (NCA) chronicles its performance audit experience according to the system-oriented approach (dating back to the 1980s) and what Supreme Audit Institutions (SAIs) can learn from that experience today, especially SAIs that want to build performance audit capacity but don't know where to begin.1
Performance audits, conforming to a system-oriented approach, focus on the proper functioning of management systems. In supporting performance audit capacity development in many countries, the NCA has seen that a system-oriented approach helps to focus the audit on structural issues, not on incidents; on the big picture, not on details; and on the performance of government and public organizations, not on legislation.
The NCA has found that financial and compliance auditors tend to think in terms of rules and regulations, and that the required change in mind-set to conduct performance audits is quite a big step and needs time.
A system-oriented approach can be of assistance in bridging both disciplines and provides a potentially more impactful way (value and benefit) as a SAI. The focus is on real, structural problems; it delivers more relevant audit results to the auditee; and it generates SAI visibility in the public sector and society.
The system-oriented approach does not focus primarily on the policy or the goals but on well-functioning management systems as a condition for effective and efficient policies.
Examples are financial management, evaluation, control and Information Communications Technology (ICT) systems.
This type of audit can use descriptive questions (which also help to detect risk areas in the system), such as:
...and can be complemented by more evaluative questions, to include:
When an audit subject has been chosen based on sound risk analysis, describing the system under audit is a good start and may also result in an interesting report of facts.
For example, in the Netherlands, programs and money flows are sometimes so complicated that a clear description of the system (preferably in schemes and chart flows) is helpful to MPs and even auditees. This is referred to as a clarification audit, and these have been very successful. Here, the audit questions have a descriptive character.
Figure 1 represents a chart flow example. It visualizes the expenses for the Netherlands railway system, as well as the responsible actors. The system complexity is evident.
An important and useful first audit step in a systemoriented approach-when it comes to a normative audit- is to ask a question related to: "What conditions must be met to ensure the economy, efficiency and effectiveness of the system?"
This conditional approach can be applied to a variety of audit subjects, such as state-owned enterprises, social security programs, a tax system and a public school. In this case, the central audit question is, "To what extent does the auditee fulfill the conditions for economy, efficiency and effectiveness?" It is interesting to identify these conditions and how these conditions can be structured to make them comprehensible to the early-stage performance auditor.
Based on NCA experience, looking from the system perspective is very helpful when defining and structuring these conditions. One of the first times the NCA chose to embark on a system-oriented approach was a government wide performance audit on subsidies management (published in 1989). The provision of subsidies was perceived as a system (see Figure 2).
This approach is further developed and generalized in Figure 3. The auditee is positioned at the top. This is very important, as performance auditors must view the situation from the auditee perspective. What must the auditee do to be efficient and effective?
Here, two main activities are distinguished:
Acting and gathering information can be aimed at a variety of subjects, either organizations (such as stateowned enterprises, a hospital or a school) or processes (such as a housing policy, social security program or tax system).
Figure 4 illustrates the organizational set-up. Starting from the bottom, the diagram outlines an entity with inputs, throughput, output and outcome.
Using a road construction company as an example (acting as a state-owned enterprise), the input would consist of raw materials (sand, cement, asphalt, but also labor). The throughput (process) is construction (of a road, which includes leveling, paving and blacktopping). The output is a road in good condition, and the outcome could be the opening up of a region that was previously difficult to reach.
In keeping with the approach as outlined in Figure 4, the minister (or the agency responsible) should act-setting goals and criteria for the road construction company- and should be informed by enterprise management about activities and results. Further, the minister or agency must take corrective actions if the information suggests cause for concern.
The system approach can also be used for processes (programs or policies), as shown Figure 5. In this instance, a well-known policy or program cycle is used. Consider, for example, a social security program. The minister or agency must prepare this program carefully by:
SAIs are in a position, facilitated by the International Organization of Supreme Audit Institutions (INTOSAI) community, to genuinely benefit from mutual experience through sharing lessons learned on implementing performance auditing. For inexperienced performance auditors, economy, efficiency and effectiveness are quite abstract concepts. Building performance audit capacity, therefore, is a challenge. However; based on the experience of the Netherlands Court of Audit (NCA), a system-oriented approach seems to be a practical first step into the challenging world of performance audit.