Technical Articles

International Journal of Auditing – April 2014


How to Increase the Impact of Environmental Performance Audits

The following article is a based on a keynote presentation by John Reed and Jean Cinq-Mars given at the 15th Meeting of the INTOSAI Working Group on Environmental Auditing held in Tallinn, Estonia, June 2013. The presentation was subsequently developed into a full Discussion Paper by the CCAF, Vérificateur général du Québec, and the Office of the Auditor General of Canada. This is a condensed version of the Discussion Paper, which can be found on the website of the CCAF at www.ccaf-fcvi.com. The presentation in Tallinn also inspired the creation of a new INTOSAI WGEA Research Project on “How to Increase the Impact of Environmental Audits,” which is being co-led by the SAIs of Lesotho and Cameroun.

Introduction

Performance auditors desire to have a positive impact on the programs and entities they audit. Similarly, performance auditors focusing on environmental, health, and safety issues want to see their recommendations implemented and programs and entities to improve as a result. Ultimately, they want environmental quality and the health and safety of all citizens to benefit from their audits.

The authors hold the view that it is possible to increase the impact of environmental performance audits and to improve environmental quality through careful audit topic selection, planning, execution, reporting, communication, and innovation. 

The foundations of a successful performance audit

Performance audits focus on a wide variety of topics. One audit will focus on a single question in a single departmental program, while another will look at several complex issues in several programs managed by many departments. Some audits focus on economy or effectiveness; others, on efficiency. Many audits examine compliance with policies, laws, and regulations while others focus on the management systems and controls that support such compliance. Most audits look at results. While environmental performance audits deal with a specialized subject, they also fall in the categories listed above.

The success of all performance audits rests on the same necessary foundations: a solid methodology, qualified people, and a sound knowledge of the subject matter.

Audit offices recognize the importance of solid methodology. Methodology that complies with professional standards, adopts best practices, and reflects key principles of quality assurance and quality control will add value and help bring about change.

To apply audit methodology as intended and complete audits in a timely manner requires strong audit teams, consisting of the right people with the right skills for each project. The most effective performance auditors usually possess a combination of key skills, particularly professional judgment, critical thinking, creativity and innovation, and the ability to lead and supervise, and to manage relationships and communications, both internally and externally.

Finally, to have an impact, auditors must select the right issues to audit, prepare a report that addresses the main questions convincingly, and communicate their conclusions effectively—all of which require a sound knowledge of the subject matter. For environmental auditors, this generally means having a good understanding of current environmental issues and of relevant environmental laws, regulations, policies, standards, and international agreements (such as those involving climate change, protection of endangered species, and waste management). It may also involve consulting advisors and specialists who have related experience with the audit topic.

Planning phase

A well-selected and planned audit is more likely to bring about significant change and add value. The following are recommended steps to take in the planning phase of an environmental performance audit.

  1. Choose topics that people connect with. All performance audits start by selecting a topic. This is the most important and often the most difficult decision in the whole process. Selecting good topics for performance audits generally requires a sound knowledge of the subject matter, a thorough risk analysis, plenty of discussions among team members and subject matter experts, and a large dose of professional judgment. Auditors can increase impact by choosing topics that legislators and the public care about, such as health, the economy and jobs, the local environment and community, or the nation as a whole.
  2. For audit objectives, focus on results, not systems. Government decision-making processes, management systems, and internal controls are important for achieving environmental results. And yet, these systems are not usually matters that people really care about or connect with. Environmental audits that focus (partially or exclusively) on tangible outcomes and results achieved are more likely to attract the attention of parliamentarians, the media, and the public than audits that focus solely on systems or procedures. Results could include the extent to which governments have solvedexisting environmental problems.
  3. For audit criteria, go beyond compliance. One of the main challenges in performance auditing is to select appropriate criteria to assess an entity’s performance in relation to the audit objective(s). Government environmental regulations are often based on minimum requirements. When auditors use minimum requirements as audit criteria, they are tacitly promoting minimum measures and are unlikely to bring about positive change. In contrast, high-impact audits are those that:
    • use best practices as expectations against which to assess programs and departments,
    • compare the performance of audited entities with similar organizations in other jurisdictions, and
    • expect to observe constant improvements over time.
  4. Regarding timing, exploit “pivot points” in the issue’s life cycle. Environmental performance auditors can take advantage of the fact that many environmental issues operate on long-term cycles that include a number of “pivot points”—key moments when decisions need to be taken that will influence future actions and events. Auditors can identify pivot points and attempt to time the conduct and reporting of an audit so that it influences the decisions that will be taken, such as before formal “meetings of parties” to consider international issues like climate change and ozone protection. Other pivot points can be the periodic review of key environmental legislation and the end of the first phase of a departmental program. It is also better to audit a potential risk area before it becomes a major disaster, such as assessing the preparedness to respond to an oil spill at sea before a large spill actually occurs.
  5. When setting the scope of the audit, consider linkages. Auditors should consider doing multiple audits on one topic in one year or over a period of years. Many environmental issues have various dimensions and are interconnected, so multiple audits on a single large topic, like climate change or biodiversity, may be a sound approach if the intent is to increase an issue’s profile and provide comprehensive coverage, which is more likely to foster debates and bring about comprehensive solutions than a narrowly focused audit. Auditors should also consider linkages to sustainable development and its three aspects (economy, environment, and society); if these can be demonstrated concretely, the resulting audit reports will appeal to a broader public and won’t be easily dismissed as supporting impractical, one-sided visions.
  6. When determining the focus, tackle the drivers of degradation. The Drivers Pressures State Impacts Responses (DPSIR) analytical framework can be a useful tool to increase the impact of environmental audits by tackling the underlying forces behind environmental degradation and impacts on people. The DPSIR framework has been used for decades as a means to understand the causes and effects of environmental problems. Each element is described below.
    • Drivers: Drivers are the overarching socio-economic forces that exert pressures on the state of the environment. Population growth, globalization, economic development, energy use, and transport are seen as particularly significant environmental drivers.
    • Pressures: Drivers in turn produce “pressures” on the environment, essentially stresses resulting from human activity. These include land use change, resource extraction, use of external inputs such as chemical fertilizers, emissions of pollutants and wastes, and the movement of organisms.
    • State: Pressures in turn affect, usually negatively, the condition or “state” of the environment, which consequently impacts human well-being and ecosystems. For example, pressures can lead to ozone depletion, climate change, pollution, and loss of biodiversity.
    • Impacts: The “impacts” ultimately lead to government interventions, or responses.
    • Responses: Auditors need to understand how the government has responded to a given environmental issue by identifying, for example, what international treaties have been signed; what policies, laws, and regulations have been enacted; and what controls and processes have been put in place. Auditors typically use these as a starting point for developing audit objectives and criteria and for auditing results achieved.

      The DPSIR framework can help environmental auditors to understand and diagnose what is happening to the environment and why, what the consequences are, and what measures the government has put in place. More importantly, using the framework may help to focus the audit on the measures taken by government to address the drivers and pressures that cause the degradation in the first place, not just on the measures to deal with the degradation. For example, an audit of drinking water quality could focus on the measures in place to treat and distribute safe drinking water and on the measures to prevent water supplies from being contaminated.
  7. Graphic:  The DPSIR Framework

    1. Conduct collaborative audits and learn from others. Environmental issues often cross borders. By collaborating with audit offices in other jurisdictions and releasing joint reports or separate reports around the same time, auditors can multiply their impact, raise the profile of an issue, and bring about new collaborative initiatives between concerned administrations to tackle common environmental problems.

    The INTOSAI WGEA has produced guidance materials for collaborative audits and has many examples of such audits on its website, such as the Coordinated International Audit on Climate Change conducted by 14 national audit offices and based on 33 individual audits. The WGEA also has a large database of audits conducted by national audit offices around the world. Between 1993 and 2011, national audit offices in over 100 countries conducted more than 3,200 financial, compliance, and performance audits related to the environment.

    Audit Execution and Examination

    A key step in audit execution and examination is to anticipate and continually assess data needs. In the planning phase, the types, sources, and limitations of evidence and data are identified. Also at this stage, auditors should try to determine the types of qualitative and quantitative data analysis they will carry out on the evidence and how it may be presented. In the examination phase, auditors need to continually assess whether the anticipated type of data and evidence is available and still relevant and if not, make any needed adjustments to the audit plan.

    The data and evidence found during the examination phase leads to observations. Performance audits, like all audits, compare a situation that exists with the way it should be, based on suitable criteria. The gaps between the two result in audit “findings” or “observations.” Examples of common audit findings include:

    • lack of compliance with rules or policies,
    • results not being achieved as intended,
    • risks not being evaluated and managed,
    • strategies not being developed or followed,
    • activities and actions of key players being poorly coordinated or having unclear roles,
    • missing data or information to measure program results or to support decisions, and
    • weak or absent oversight.

    To answer the burning question, “Why do these deficiencies occur?” auditors should perform root cause analysis, which can support effective recommendations that lead to solutions that prevent the problem from recurring. One caution, however: root causes that are traced to the merits of policy, availability of resources, or partisanship can be difficult for legislative auditors to address.

    Reporting Audit Findings

    By the end of the examination phase, auditors have usually gathered evidence from various sources, all of which has to be considered, analyzed, and retained or discarded. In this evidence lies the story the audit will eventually tell. The challenge is to determine what the main messages will be and how to best present them in a clear, convincing manner.

    The most effective reports answer the following questions:

    • “What?”—Identify the problems uncovered by the audit.
    •  “So what?”—Explain why the reader should care about the audit findings.
    • “Why so?”—Identify the root cause of problems or observations.
    •  “What next?”—Highlight the recommendations or solutions proposed.

    In addition, auditors should strike the right tone and communicate a balance of positive and negative findings that reflects the evidence gathered.

    Auditors should use the roll-up technique for developing good audit reports. This technique involves filtering information and prioritizing messages for inclusion in the report. Auditors examine the large amounts of information normally collected during an audit in order to identify usable evidence and include in the report only those facts, observations, and conclusions that are material, significant, and/or of high risk. In the environmental domain, significance and risk can and should relate to human health, ecosystem functioning, and the financial consequences of environmental degradation.

    Auditors should use their reports to educate. Environmental questions are often complex and it may be necessary to explain important concepts in a report’s introduction to help readers fully understand audit findings and their significance. The introduction is an ideal place to provide basic context and background information and make links to any related economic and social aspects.

    Report authors should use plain language and avoid being too technical. Similarly, reports that help readers identify with the topic and care about the audit findings will have more influence. This can be achieved by using concrete case studies, such as ones about potential health impacts in urban areas.

    Ultimately, the impact of an environmental performance audit will depend on the quality of the recommendations and on their implementation. Too often, recommendations are prepared at the end of the audit, seemingly as an afterthought. Instead, the thought process for recommendations should start during the examination phase.

    Auditors can make strategic recommendations by focusing on the “pivot points” (discussed earlier) in relevant decision-making processes.

    They can also aim to create a “domino effect.” The domino effect refers to situations in which changes to one element of a system trigger changes in its other elements; the effect is greater when system elements are closely interlinked and when recommendations are targeted at a key point in the decision-making process. For example, introducing a carbon tax will create a domino effect in society and in the economy since the tax is likely to result in reduced consumption of carbon-intensive products, lower greenhouse gas emissions, increased use of public transport, better air quality, improved health for citizens, and lower health care costs.

    Recommendations that are superficial (for example, “the entity does not have a strategy, so we recommend it develop a strategy”) or superfluous (for example, “the entity should continue to do…”) are unlikely to lead to significant changes. In some cases they may be required as a first step, but to be effective, recommendations should:

    • be strategic, not operational, in nature;
    • address the root cause of identified problems, not their symptoms;
    • focus on the expected result or outcome, not on the means of getting there;
    • try to be innovative and to push the boundaries (such as by analyzing and comparing the practices used in other administrations); and
    • always remain realistic and consider the views of the audited entity on the proposed recommendations.

    Another way to increase the impact of an environmental performance audit is to reach out to audiences beyond the usual suspects that can help to reinforce audit messages and impact. For example, environmental auditors can contact specialized media, journals, academics, and members of civil society, including grassroots organizations and youth.

    Follow-up

    If one audit can have an impact, two audits can have even more. Conducting one or a series of environmental audits and later conducting a follow-up audit to determine progress in resolving deficiencies and implementing recommendations can ensure that audit work will have a sustainable impact. Indeed, when entities know that a follow-up audit might (or will) take place, they are more likely to take concrete actions.

    Follow-up audits usually occur a few years after the original audit—enough time for entities to implement recommendations. In most cases, only one follow-up will be done. But sometimes it is good to have a long-term plan for more follow-up audits, especially in the environment domain, where issues tend to have long time frames.

    Conclusion

    Governments are tasked with managing and addressing key environmental issues, from the impacts of climate change to urban smog. Audit offices have the mandate to assess the implementation and effectiveness of the management of these issues and many more.

    Environmental audits can affect change in these areas. But this requires careful attention to audit topic selection, planning, execution, reporting, and communication.

    By focusing on these elements, auditors can increase the impact of environmental performance audits and, as a result, improve environmental quality through better and more effectively managed programs. Even auditors play a role in the concept behind the saying, “We don’t inherit the Earth from our parents— we borrow it from our children.”

    Acknowledgements

    The authors would like to thank the following individuals for their contributions to this paper: Pierre Frechette, Kimberley Leach, Neil Maxwell, George Stuetz and Scott Vaughan.