Systematic Approaches to Performance Audits of Environmental Policies
By KIM Burmshik, Deputy Director, and JEON Hyeong Cheol, Director, The Land and Maritime Affairs Audit Division III, The Board of Audit and Inspection
Introduction
Auditing the performance of government environmental programs is concerned with the three E’s, Economy, Efficiency, and Effectiveness, like most performance audits. However, it is very difficult to set up performance indicators and select adequate methodology to analyze the performance results for auditing in various environmental programs.
In this paper, a performance audit case of an environmental policy conducted by the Board of Audit and Inspection (BAI), the supreme audit institution of Korea, is introduced to share with INTOSAI advisable ways of systematic approaches to performance audit.
Background
The Korean government’s budget allocated for managing fine dust, defined as particulate matter under 2.5μm diameter (PM2.5), was 491.7 billion KRW (approx. 491.7 million USD) in 2016. But in 2019, it was dramatically increased to 2.17 trillion KRW (approx. 2.17 billion USD). However, there was still a concern over the effectiveness of government programs managing fine dust across the whole country, and some even pointed out that the effectiveness of the detailed measures of the program was not being analyzed scientifically and sufficiently.
Audit Design and Approach
In Korea, the central government establishes a comprehensive management plan for fine dust based on the research results regarding the current contamination status by fine dust in the ambient air. Then, local governments establish their own plans for implementing the central government’s plan in their regions, execute allocated budgets accordingly, and report the results of their implementations regularly to the central government, mainly the Ministry of Environment (MoE).
The plan of the central government is established in the following manner: the central government (1) analyzes the current status of fine dust concentration in the ambient air, and evaluates the contribution ratio of each air pollutant emission source to the concentration; (2) sets the goal of fine dust concentration level and the reduction amount of total air pollutants needed to achieve the aim of fine dust concentration in the target year; and finally, (3) establishes implementation plans for reducing the emission amounts of each pollutant source to meet the required total reduction amount of air pollutants.
Therefore, in order to oversee the government’s implementation of its plan for fine dust, it is a requisite for BAI auditors to be able to inspect scientifically what the pollution sources are and examine systematically whether the implementation plans are effective and efficient. To this end, BAI auditors approached this issue in the following ways.
First, the auditors took a linear approach. They reviewed the whole process of the policy cycle step by step from design, execution, to performance evaluation, to see how each step was implemented as time progressed. This approach can also be interpreted as a vertical analysis, in the sense that it analyzes the implementation plans of all parties, from the most top-tier level to the working-level, and how the plans were implemented and checked by them.
To make this approach successful, BAI auditors: (a) probed scientifically the accuracy of evidence materials used for establishing the comprehensive plan of MoE, in close cooperation with expert groups, including academic societies; (b) evaluated the actual implementation of each level’s plan through interviews with the public officials in charge of implementing the plans; (c) identified problems and difficulties in the plans through exchanging feedbacks with the public officials; and (d) took a survey to realize citizens’ awareness on fine dust management issues, held workshops with non-governmental organizations to understand how the public policies for managing fine dust were being thought by the society, and then utilized the results to set up the main focus for this audit.
Second, the auditors took a point analysis approach. It was to identify the difficulties, if any, in the process of implementation to examine the effectiveness of each fine dust policy, and to find solutions for improvement. This approach can also be interpreted as a horizontal analysis, in the sense that all stakeholders of fine dust policies should gather to give their opinions on the implementations of government policies and share the performance of the policies.
To make this approach successful, BAI auditors: (a) conducted a poll among the public officials of central and local governments on the structural difficulties with implementing each policy and possible ways to improve them; (b) held workshops with the public officials to hear about structural difficulties with implementation of fine dust policies and possible solutions; and (c) organized two workshops, one with businesses with air pollutant-emitting facilities and the other with specialized environmental construction businesses, to identify the problems around the policies of the MoE found by them on-site, and hear from them whether the problems can be overcome using the current available technology. Based on the results of these efforts, the main points to check in this audit were established.
In order to set up a questionnaire for performance auditing using these two approaches, BAI auditors analyzed the progress of the policy cycle, from design to execution, to identify potential risks of each phase of the policy cycle. The questionnaire was prepared to: (a) secure the accuracy and timeliness of background data used in the public policies for managing fine dust; (b) examine the reliability of the anticipated effects of government plans; and (c) look into the basis of performance evaluation used in executing the plans and check whether any supplementary plans were established to improve flaws or problems of implementation plans.
Audit Results and Recommendations
As a result of auditing the government’s policies for managing fine dust with the aforementioned two approaches, it was found that it is still difficult for ordinary citizens to feel the effectiveness of the government policies for reducing the concentration of fine dust in the air, because even if the government spends an immense amount of budget in controlling the domestic situation, there were still limitations, as 50% of the fine dust comes from abroad.
In this audit, it was demonstrated that the MoE applied correct data of emission amounts of pollutant sources in establishing their plans for reducing fine dust. It was as accurate as the cases of other countries. However, there were some flaws in certain parts of the government’s plans. For example, some pollutant sources were omitted, some target amounts of emission were under-estimated due to the inaccuracy of source data; some of the individual implementation plans were executed without verifying the effectiveness of the plan; and it became difficult to expect effectiveness of some implementation plans due to the lack of proper follow-up management. Details of these problems are shown in Table 2.
Lessons Learned and Conclusion
In conducting performance audit of government environmental programs, it is useful and proper to employ both the linear approach and point analysis approach. The linear approach enables auditors to analyze each phase of the policy cycle, whereas the point analysis makes it possible to analyze the effectiveness and efficiency of each implementation program.
When employing these approaches in auditing, it is important to keep in mind that auditors should have full knowledge of given audit subjects, engage external experts directly and indirectly in the process of audit, and identify problems based on scientific analysis to establish alternative solutions.
