Technical Articles



International Journal of Auditing – October 2004

Who Audits the Auditor? The International Peer Review of the Office of the Auditor General of Canada

A fundamental goal of the Office of the Auditor General of Canada (OAG) is to promote good governance in Canada’s federal government departments and agencies. Following through on its commitment to good governance in its own work, the OAG underwent an external review of its financial audit practice in 1999. Auditor General Sheila Fraser took the next logical step in 2002-2003 and formally committed the OAG to undergo an external review of its performance audit practice by 2005. This article summarizes the process that was followed to carry out that review.

The purpose of the review was to assess the extent to which the OAG’s performance audit practice reflects recognized professional standards and is operating effectively to produce independent, objective, and supportable information that Canada’s Parliament can rely on to examine the government’s performance and hold it to account.

The OAG’s peer review was the first time the legislative audit practice of an SAI was assessed by a team of its international peers. The review was carried out in accordance with commonly accepted auditing principles over the course of a year by a team led by the United Kingdom’s National Audit Office with representatives from the SAIs of Norway, the Netherlands, and France. The U.S. General Accounting Office (now the Government Accountability Office) participated as an observer and prepared a lessons learned report on the process.

Preparing for the Review

Planning for the peer review began in earnest in late 2001, led by the OAG’s Professional Practices Group. The first step was consolidating the observations and findings of all recent reviews, reports, and action plans related to the OAG’s performance audit practice and updating the status of their recommendations. Lessons learned from the 1999 external review of the financial audit practice were documented, and the external review practices of several SAIs were reviewed to identify good practices.

In January 2002, the results of that work were presented to the OAG’s Executive Committee, which recognized that all of the office’s committees concerned with the performance audit practice would have to be involved in preparing for the review. The most important committee was the Practice Development Committee, which is responsible for approving policy, guidance, and methodology related to performance audits. The Audit Committee was assigned an oversight role, and the Auditor General’s Panel of Senior Advisors provided independent advice. A small internal working group with membership from across the office was also created to keep employees informed and to provide guidance and advice as the preparations progressed.

On the advice of her panel, the Auditor General decided to extend the review to all performance audit work included in her periodic reports to Parliament for 2003, with the external review team reporting back to her early in 2004. That decision effectively limited the preparation period to 8 months.

The Executive Committee agreed that the reviewers should assess the design and operation of the quality management framework (QMF), the policies and procedures that govern the OAG’s performance audit practice. The next step was to determine what policy and practice changes were needed to be “review ready” by January 2003. The OAG recognized that because continual improvement is a basic part of the quality management process, the review could not wait for the audit practice to reach a “steady state.” However, it also understood that some improvements should probably be accelerated. The Professional Practices Group asked the OAG’s Assistant Auditors General for their input and, in June 2002, presented the major opportunities for improvement to the Executive Committee. A proposal to develop and implement these changes was approved in July, followed in August by an implementation plan outlining the necessary methodology projects, internal practice reviews, consultation, and training.

A communications plan was developed to inform and engage employees at all phases of the review. As preparations for policy and methodology changes moved into high gear, communications were targeted to the audit practitioners who would be affected most directly by the review. This included e-mails to alert them of pending changes to the performance audit manual, practice advisories to provide guidance as changes were approved, training sessions, and events such as the OAG’s annual performance audit symposium.

In September 2002, the Professional Practices Group undertook an internal practice review “blitz” to assess adherence to policy, identify opportunities for improvement, and provide the Auditor General with assurance that the performance audit practice was robust.

At the direction of the Executive Committee and concurrent with the internal review and methodology initiatives under way, the OAG engaged consultants to carry out several separate examinations of the office’s performance audit policies and practices. The objective was to get an outside opinion to validate whether the OAG’s own assessment of its policies and practices was accurate.

In this initiative, the offices of the Auditors General of British Columbia and Quebec each completed a review of OAG performance audits. The OAG also contracted with an outside consultant to carry out a gap analysis of its QMF relative to the Quality Management Standard promulgated by the International Organization for Standardization (ISO). Finally, the London School of Economics’ Enterprise LSE Ltd. was engaged to evaluate two of our products using their criteria for performance audit reports.

Preparations for the peer review concluded with the launch of a revised performance audit manual in December 2002. The revised manual represents the culmination of an intensive period of action that was sustained by the leadership of the Executive Committee, the hard work of the Professional Practices Group, and, most importantly, the strong support of practitioners and audit services employees across the office.

In December 2002, the Auditor General met again with Parliament’s Public Accounts Committee to brief its members on plans for carrying out and reporting on the peer review. The Committee expressed its support for the peer review process and, in particular, for the Auditor General’s plans to publicly report the results.

Establishing the Ground Rules for the Review

In February 2003, the OAG asked members of INTOSAI if they would be interested in participating in a peer review of the OAG’s performance audit practice. There were a number of important reasons why the Auditor General decided to engage the international audit community and to adopt a peer review model for the external review of the OAG’s performance audit practice.

First, it was considered critical that the review team be free and appear to be free of any conflict of interest or other impairment to its independence. This ruled out a fee-for-service arrangement, such as a contract with a local consulting or accounting firm. The option of forming a team of representatives from Canada’s provincial auditors general was also ruled out since several of their audit offices participate with the OAG in other endeavors, including practice reviews.

Independence, competence, and credibility were prerequisites for the review team and the process.Second, it was essential that those engaged in the review be competent to perform the necessary work. Many members of the international audit community have mandates similar to the OAG’s and have been engaged in performance audit work for decades. SAIs understand the mechanics of legislative audit at the national level and possess the knowledge, experience, and skills required to carry out the work. Finally, it was important that the approach adopted for the review be widely understood and recognized as credible.

A Focus on Independence, Objectivity, and Reliability

Within a few weeks, several SAIs had expressed interest, and in late February 2003 the group met to discuss how a peer review of an SAI could be carried out.

It quickly became apparent that other SAIs might also be interested in undergoing a peer review. Early discussions therefore centered on values and principles that SAIs share and that could form the basis of a common approach to peer reviews of SAIs.

After considerable discussion, the peers agreed that a principles-based approach focusing on independence, objectivity, and reliability should be acceptable to all SAIs contemplating a peer review. Beyond those shared principles, it was agreed that SAIs should be assessed against the standards of professional practice applicable in their home countries and that peer reviews should be carried out in accordance with commonly accepted auditing principles consistent with INTOSAI’s Code of Ethics and Auditing Standards.

The peers also agreed that SAIs should be free to specify supplementary criteria for their own peer review should they wish to do so.

The peers agreed that such reviews should: examine both the design and the operation of the SAI’s audit practice; focus on key controls related to independence,objectivity, and reliability; use the standards of professional practices applicable in the SAI’s own country; and be carried out in a collegial manner so as to maximize learning for all concerned.Discussions progressed to the practical considerations of performing peer reviews, including details such as the level of effort involved, leadership, methodology, objectives, scope, and criteria.

The peers agreed it was crucial that peer reviews be tightly focused and professionally executed so that the level of effort and the financial resource requirements would not be unnecessarily burdensome to either the reviewers or the reviewed SAI. If the peer review process were to prove too burdensome, the appetite for performing or undergoing one would quickly fade. It was agreed that an SAI should request a peer review only after a thorough self-assessment demonstrated the audit practice and the organization to be review-ready.

The peers also agreed that some advance work could be done to manage the level of effort involved. First, peers could examine the documented management framework underpinning the SAI’s audit practice to determine whether obvious gaps existed between the documented practice requirements and the key principles of independence, objectivity, and reliability or the standards of professional practice applicable in the SAI’s country. The peer reviewers could carry out a design assessment relatively quickly at their respective home offices to identify matters that the SAI should address before undergoing a full peer review.

Second, the SAI’s internal practice review function could be assessed to determine whether the peer review could rely on it to provide assurance of the SAI’s compliance with applicable professional standards. If a high degree of reliance was indicated, the peer review could then focus on whether critical processes and controls that support the fundamental attributes of a good performance audit practice––independence, objectivity, and reliability––were operating effectively.

The peers also agreed that an important aspect of peer reviews would be the learning experience they afford for all concerned. It was agreed that peer reviews should be carried out in a collegial manner, with open and transparent communication between the review leader and the reviewed SAI throughout the process. There should be no surprises.

Thus, the peers agreed on an open, transparent peer review process with three key elements:

  • design assessment—to determine whether the management framework underpinning the SAI’s performance audit practice reflects applicable professional standards and appropriate measures to ensure that the products of the practice are independent, objective, and reliable;
  • reliance assessment—to determine whether the SAI’s own practice review/inspection function provides assurance that the practice is operating in compliance with the SAI’s policy expectations and applicable professional standards; and
  • implementation assessment—to determine whether the performance audit practice is operating effectively to provide users with independent, objective, and reliable information.

Assembling the Team and Performing the Peer Review

The peers met again in April 2003 to plan the necessary work for the Canadian peer review including ls of the approach, process, and administrative arrangements. In May 2003, the OAG and the review team signed a memorandum of understanding to officially launch the process. Each participating SAI agreed to assign two senior-level performance auditors to the review team. As the reviewed SAI, the OAG agreed to cover the costs of the team’s travel and accommodation in accordance with its own policies and regulations.

The first component of the review—the design assessment—began in June 2003. The peer review team examined the OAG’s quality management framework (QMF). The team examined the direction available to audit practitioners to determine whether it reflected applicable standards of professional practice and the legislative authorities in the Auditor General Act. The team also looked for mechanisms in place to ensure due regard to economy and effectiveness in the conduct of the OAG’s work.

The reliance assessment also took place in June. It focused on the OAG’s practice review function to determine whether it could be relied upon to provide assurance of compliance with the QMF.

The implementation assessment began in early July, when the review team selected the first sample of performance audit files to examine. The implementation assessment focused on whether the QMF for the performance audit practice operated effectively to meet its goals of producing independent, objective, and reliable information and ensuring that the work is carried out with due regard to economy and effectiveness.

During the implementation assessment, the peer review team met three more times over 5 months to compare notes and discuss preliminary findings and recommendations. Members also communicated regularly by e-mail and telephone.

In November 2003, the peer review team spent 2 weeks at the OAG’s Ottawa headquarters conducting interviews and focus groups with performance audit practitioners, functional area leaders, subject matter specialists, and senior executives. The discussions were intended to confirm the peers’ understanding of the key elements of the QMF and to verify that management and staff clearly understood their roles and responsibilities in its effective operation.

The peer review team presented its initial findings to the OAG’s Executive Committee
at the end of November 2003 and completed its report in February 2004. The team concluded that the OAG’s performance audit practice was suitably designed and operating effectively to achieve its objectives.

Responding to the Peer Review Report

The peer review report and the action plan the OAG prepared to address the suggestions for improvement were tabled at a meeting of the Public Accounts Committee in March 2004. They are available on the OAG’s Web site (www.oagbvg.gc.ca).

In developing its action plan, the OAG identified senior-level “champions” with leadership responsibility for each action item. The OAG’s Executive Working Group for Strategic Planning assumed oversight responsibility for implementation of the action plan and called for quarterly progress reports on the status of each action item. Overall progress on the action plan will be reported periodically to the OAG’s Executive Committee and publicly in the OAG’s annual performance report.

Lessons Learned

The peer review of the OAG’s performance audit practice demonstrated that an international team can perform an external review that will satisfy appropriate standards, provide value to the reviewed organization, and afford a learning experience for review team members and the reviewed SAI alike.

However, the decision to undergo a peer review should not be taken lightly. The review and preparations for it require careful consideration and management by both the reviewed SAI and the review leader. SAIs contemplating a peer review should note that the process requires a significant investment of financial and human resources for the international peers involved and for the reviewed SAI. The OAG has estimated that the peer review of its performance audit practice cost about Canadian $800,000 over the 2-year period.

Still, the OAG believes the investment in peer review was worth the effort. It is pleased with the new relationships the process has fostered and the overall outcome. In its report to the House of Commons, the Public Accounts Committee said the following:

Having independent and external confirmation of the soundness of the
Office’s value-for-money practices provides additional assurances about the
integrity of the accountability process of the Office of the Auditor General
of Canada and adds credibility to its efforts to provide objective,
supportable and reliable information on the administration of government
programs and activities.

The Committee wishes to emphasize the excellent work that the Auditor
General and her Office consistently provide in supporting all
Parliamentarians in their efforts to hold Government to account. Canadians
can take a great pride in knowing that the Office of the Auditor General has
set and maintained high standards of professionalism and dedication to its
responsibilities.

That was exactly the result the OAG had aimed for.

For more information, contact the author at fergusja@oag-bvg.gc.ca