The Impact of Performance Auditing: A Practice Friendly Review

by Mr. Vital Put, Councilor, Belgian Court of Audit, and Visiting Professor, Public Governance Institute, Catholic University of Leuven

High Expectations—Impact of Supreme Audit Institutions

The International Organization of Supreme Audit Institutions and the United Nations (UN) have created high expectations for Supreme Audit Institutions (SAIs), as evident in several INTOSAI standards and UN resolutions.

Additionally, the Organisation for Economic Co-operation and Development (OECD) expects SAIs to contribute to better governance,¹ and many development aid agencies consider SAIs to be “ambassadors and promoters at the international level of concepts, such as transparency, accountability, governance and performance.”²

In many strategic plans and annual reports, SAIs view themselves as engines for change. For example, the Belgian Court of Audit intends to “maximize the impact of audits on government policy and government functioning, on the management of public resources and on parliamentary supervision.”³

With such high expectations, the question arises, “To what extent can SAIs actually contribute to a better government?”

High Expectations—Can They Be Met?

SAI performance measurement systems provide interesting information; however, since this information is produced by the SAIs themselves, one could argue that this information is biased. Because scientific research is more objective, this article will use academic literature as its primary source of data. Thirty-one studies were found (1980-2017) that specifically discuss the impact of performance auditing and SAIs⁴, with surveys, case studies or statistical analyses representing the most common research designs.

In most studies, the term “impact” is defined in an instrumental way, such as the implementation of recommendations and changes in management practices, policy measures, etc. There are some studies that also consider conceptual impact to include creating new insights and stimulating discussion. Yet, there are dimensions, such as tactical use, that receive little attention.

Research has shown that performance audit provides a moderate positive impact (instrumental as well as conceptual) on audited organizations. The literature describes nine groups of factors that contribute to performance auditing impact, some of which relate to the audit process and can be influenced by the auditor, including:

• Constructive relationship between auditor and auditee during the audit (a setting characterized by trust, open dialogue, shared values and ideas);
• Audit report quality (relevant findings, clear audit criteria, feasible recommendations); and
• Follow-up of recommendations.
• While auditor impact heavily depends on actions and reports, some factors can only be influenced by a SAI to a limited extent, including the:
• Audited entity’s willingness and culture (openness to learn from feedback, criticism);
• Coincidence of project reform with audit results;
• Chance events, such as change in management;
• Media pressure;
• Civil society pressure; and
• Parliamentary pressure (when members of parliament pay attention to performance audit, changes recommended by the performance audit report are more likely to be adopted).

According to one study, “willingness of the auditee” and “pressure from parliament” are interchangeable aspects: if auditees agree with the auditor’s findings, they will implement change, regardless of whether the report triggers parliamentary debate; if they disagree they will resist change, unless there is pressure from parliament.

Several studies offer support for the hypothesis that strong vertical accountability—between citizens and politicians; through a competitive electoral process; through civil organizations; through the news media—is an essential condition for strong horizontal accountability, including checks on government by other public entities, such as SAIs).

Consequently, it does not seem plausible that SAIs could be “pockets of effectiveness”⁵ when they are embedded in a political system that lacks a potent parliament or robust rule of law.⁶

Impact of Performance Audit—Existing Research Blind Spots

Existing research offers interesting insights into performance auditing and SAI impact. Nevertheless, there are important blind spots. While the scope of this article does not allow for an exhaustive analysis, two important limitations must be noted: (1) some factors remain out of sight, and (2) there is too little regard for the complexity of causal relationships.

There are various elements that could influence performance audit impact where studies are scarce or nonexistent.

• The relationship between the nature of the performance audits and impact. Performance audits can be very heterogeneous, even within one SAI. A link between the nature of the conducted performance audits and the degree (and nature) of its impact is probable. For example, does an audit focused on processes lead to greater impact than an audit focused on effectiveness?
• Public service delivery variances. Public services can be delivered through different instruments, such as publicly owned entities or quasi-market mechanisms. Can performance audit be equally effective in all administrative contexts? Is performance audit still useful when citizens can give incentives to perform effectively (as in the market model)?

Current research hardly addresses various causal paths that can lead to impact. For example, is a certain factor (a constructive relationship between auditor and auditee; pressure from the media) a necessary condition (along with other factors) for impact, or is it only a contributing factor?

Unraveling such complex causal relationships requires an adequate research approach, such as Qualitative Comparative Analysis (QCA).

QCA assumes a configurational view of causality whereby different combinations of factors can lead to impact, and each factor may have a different effect depending on the combination.

Conclusions

Overall, research demonstrates performance auditing to be a useful activity that contributes to better government. Factors revealed in this literature review are remarkably consistent with good practices recommended in INTOSAI’s guide on “How to Increase the Use and Impact of Audit Reports: A Guide for Supreme Audit Institutions.”

Additionally, many of the suggested practices found in the guide perfectly match scientific research findings, including the following (parenthetical text refers to phrases found in the guide):

• Constructive relationship between auditor and auditee during the audit (inform the auditee of what to expect during the audit; write clear and understandable audit reports).
• Quality of the audit report (choose topics that are useful for legislatures and auditees, ensure that quality is built into the audit process; write recommendations that lead to change).
• Follow-up of recommendations (to reveal if progress has been made).
• Media pressure (consider the interests of other stakeholders; help the media understand the audit reports).
• Civil society pressure (consult stakeholders; use civil society organizations to increase reach).
• Parliamentary pressure (know that the legislature is one of the SAIs most important clients).

The available scientific research thus strengthens the credibility of INTOSAI guidance. The research and INTOSAI guidance offer good points of departure for SAIs that aim to strengthen impact.

References

¹OECD (2015), Supreme Audit Institutions and Good Governance. Oversight, Insight and Foresight, OECD Publishing, p. 141.

²Dutzler, B. (2013), Capacity Development and Supreme Audit Institutions: GIZ’s Approach, in: GIZ & INTOSAI (eds), Supreme Audit Institutions. Accountability for Development, pp. 51-68, Baden-Baden: Nomos, p. 281.

³Rekenhof, Jaarverslag 2014, 2015, pp. 12-15.

⁴Apart from studies that deal with the impact of performance auditing, a few articles about the impact of SAIs in general were included, because they also reveal a number of factors that are important for the impact of performance auditing. This article builds upon an earlier review of studies from the period 1980–2008, published in: Van Loocke, E., Put, V. (2011), The impact of Performance Audits of SAIs: A Review of the Existing Evidence, and in: J. Lonsdale, P. Wilkins, T. Ling (eds.), Performance Auditing: Contributing to Accountability in Democratic Government: pp. 175-208. For information about the reviewed studies, please contact the author via email at PutV@ccrek.be.

⁵Leonard, D.K. (2010), Pockets of effective agencies in weak governance states: where are they likely and why does it matter? Public Administration and Development, pp. 30, 91–101.

⁶As stated in the foreword of the Lima Declaration (International Standards of Supreme Audit Institutions ¹): the rule of law and democracy are essential premises for independent government auditing.