How do Supreme Audit Institutions measure their impact? Results from the Annual Meeting of the INTOSAI Working Group on Evaluation of Public Policies and Programs (WGEPPP)

By Andrea Haeuptli – representing the Chair of the Working Group on Evaluation of Public Policies and Programs

The existence of Supreme Audit Institutions (SAIs) is vital in order to assure governmental accountability as well as the efficient and effective functioning of the State. Especially through policy evaluation and performance audit, SAIs go beyond the assurance of correct and compliant administrative activities. They contribute to effective policy implementation and as such, to the provision of societally crucial goods and services. This can only take place when SAIs are effective themselves. Furthermore, in order to be accountable, they ought to provide the public with an assessment of their performance and the ultimate impact of their work.

Yet, the very measurement of SAIs’ impact and performance can be challenging. The institutions cover the extensive portfolio of all state activities, performing various types of audits, evaluations and juridical tasks. They often lack a comparable benchmark of performance given the individuality of each SAI and their context.

During the annual meeting of the Working Group on Evaluation of Public Policies and Programs, 34 participants from 19 SAIs tackled this tricky topic. In a preliminary survey, the WGEPPP Presidency aimed at getting insights into the status quo of current impact measurement of SAIs around the globe. A total of 22 member SAIs[1] provided answers to this survey. Its results are presented in the following.

Output documentation and recommendation controlling

A straightforward performance measure are statistics related to the output of SAIs. 77% of responding SAIs indicated that they publish the number of carried out audits per year. 86% include the number of published reports per year. Fewer SAIs (41%) disclose the number of planned audits per year. Nevertheless, the information on discrepancies between planned and performed audits could be indicative for the performance of the given SAI.

When it comes to information on recommendations, 72% of respondents indicated to track the status of recommendations, based on documentation and follow-up audits assuring the actual implementation. Yet, only 45% of respondents stated to publish the number of accepted / implemented recommendations per year. In further contrast, only 14% do also publish the number of recommendations which are refused / not implemented by the auditees. Meanwhile, recommendation tracking is generally performed; however, this information does not as often reach SAIs’ stakeholders beyond the auditees.

Calculations of financial savings

27% of responding SAIs stated to measure the amount of potential savings based on their audits and publish these numbers. In 83% of these cases, the measure consists in a “value for money” indicator, i.e., stating that a given amount of SAI budget equals a given reduction in state expenses. The ways of calculating these measures vary widely. They consist for instance in the ratio of expected benefits vs. expected costs (SAI Thailand), potential benefits vs. effective benefits (SAI Brazil) or in the correlation between overall SAI budget and reductions in state budget (SAI Latvia).

Auditee surveys

The auditee is a main source of information when it comes to the performance of SAIs. Although subjective, their assessment of improvements and side effects of SAIs audit activity is crucial. 41% of respondents stated to conduct surveys among auditees. In one third of these cases, the survey constitutes a standard procedure performed for each audit. In 80% of cases, the survey is focused on a specific audit and its results rather than an overall assessment of the SAIs work.

Civil society and political discourse

The results of SAIs’ activities have an impact beyond auditees and state administration. They can be crucial for civil society and political discourse. Accordingly, 46% of responding SAIs track media coverage based on their reports. 41% include measurements of website traffic and social media interaction. Nevertheless, those numbers are not directly integrated into general impact measurements of the responding SAIs. They figure as indicative number in activity reports and do inform SAIs on the public and political perception of their work on a qualitative level. In addition, 50% of responding SAIs actively seek feedback from citizen panels, working groups and NGOs. When it comes to interactions in Parliament, 50% of responding SAIs indicated to measure the proportion of reports commissioned by Parliament and / or Government. 36% state to measure how often audit reports are cited in Parliament. The same percentage measures how often the Parliament / Government introduces legislative changes due to audit reports.

Peer reviews and external assessments

36% of responding SAIs indicated that they have already undergone assessments of their impact by external actors. In most cases, this encompasses peer reviews performed by other SAIs.

Conclusions

The presented figures show general trends in how SAIs assess their performance. It becomes clear that mere output measures provide little insight into the actual effectiveness of SAIs’ work. The lack – or non-existence – of a general benchmark for all SAIs makes it difficult to understand the meaning of such results. Accordingly, the figures remain also difficult to interpret for the general public.

Some SAIs thus widen the spectrum of measurements to more explicit indicators such as value for money measures. Yet, there is also a need for qualitative information. Civil society and political discourse are of interest for SAIs as well, but rather on an informative level and is typically not included in performance indicators.

The range of possibilities and practices for measuring a SAIs performance vary thus largely. The discussion of these results during the annual WGEPPP meeting showed that member SAIs would be interested in a standardization of performance and impact measures. At the same time, the discussion made clear that one standard for all SAIs could easily water down to the smallest common denominator and therefore provide little insight. The Working Group on Evaluation of Public Policies and Programs (WGEPPP) will follow up with this important topic in the future.

[1] Responding SAIs include: Algeria, Brazil, Bulgaria, Denmark, European Court of Audit, Finland, France, Italy, Kenya, Latvia, Lithuania, Madagascar, New Zealand, Pakistan, Peru, Philippines, Slovakia, Spain, Sweden, Switzerland, Thailand, United States