Highlights

International Journal of Government Auditing – January 2013


Designing and Implementing a Quality Management Framework in the SAI of Pakistan

The Office of the Auditor-General of Pakistan is committed to producing high quality audit reports and contributing to improved service delivery in the public sector. Driven by this vision, we developed our Financial Audit Manual (FAM) based on International Auditing Standards (IAS) and introduced it in 2006. The manual advocates the use of a risk-based methodology in public sector auditing and provides quality checks at all stages of an audit cycle. We revamped our training programs and developed and diffused knowledge products to embed the FAM in our practices[1] to enable auditors to use the FAM effectively in their work.

At that time, we had a Central Quality Assurance Committee (CQAC) to ensure quality in our products. The CQAC’s mandate was limited to reviewing, improving, and finalizing audit reports of field audit offices (FAO). The CQAC reported that introducing the FAM had improved the quality of our audit reports to a certain extent. However, other quality-related problems remained.

Using benchmarking techniques, we followed a structured participative approach in developing and institutionalizing our quality management framework (QMF) in 2010. It uses prescribed standards as benchmarks and examines the extent of FAO compliance with these benchmarks in audit processes and reports. Although the QMF has been operative only for about 2 years, it has already improved quality of our audit processes and reports. This article shares our experience in designing and institutionalizing our QMF.[2]

Elements of Our QMF

SAIs should establish a quality management system that responds to the specific challenges they face. The system should help them test the existence, relevance, adequacy, and operational effectiveness of quality controls within the SAI.[3] In addition, it should provide mechanisms to not only identify and fix gaps in compliance with prescribed audit standards but also locate bottlenecks requiring changes in policy, systems, and procedures. The system should allow different tiers of an SAI to make meaningful contributions toward quality improvement.

We integrated the requirements of the FAM and sectoral guidelines[4] in the design of our QMF. In addition to ISSAI 40, we also factored in relevant EUROSAI and ASOSAI standards and guidelines. As intended, our QMF is consistent with the system of quality management envisaged in the preceding paragraph. It captures quality at all stages of an audit cycle and acts as an effective decision support system for quality management, as explained below:

Quality Assurance Mechanism: This mechanism provides reasonable assurance that our personnel comply with the professional standards and that our reports are appropriate in the circumstances. The QMF puts the responsibility for quality assurance processes with the heads of FAOs. The processes include updating the permanent file, preparing the planning file, reviewing audit planning and audit completion checklists, filling in the “Post-Audit Quality Assurance Checklist,” maintaining documentation and ensuring the completion of audits within the planned timelines, and reviewing quality assurance and quality control reports. The QMF specifies roles and responsibilities of heads of FAOs, audit wings, and other officials in these processes. Periodic quality reviews track the extent to which FAO officials have taken the required actions. Standardized templates/checklists are included in the QMF to facilitate reviews.

Quality Control Mechanism: This mechanism checks the effectiveness of the quality assurance mechanism in the FAOs and evaluates and grades FAO audit reports. An Audit Quality Management Wing (AQMW) was set up to perform these tasks. It is headed by a senior quality management specialist (SQMS) who supervises the work of a team of quality management specialists (QMS). One QMS has been placed in each of our six audit wings and reports to the SQMS. The AQMW is at sufficiently higher level within the SAI to work independently. It is staffed by experts from the marketplace, but we plan to gradually replace them with our regular officers.

  • Periodic Quality Control Reviews: The SQMS prepares quality control review (QCR) programs covering a specific phase of the audit cycle.[5] The SQMS notifies the FAOs of the scope and dates of the QCR and the name of the QMS who will conduct it. Each QMS randomly selects audits for the QCR. For each audit selected, the QMS compares the steps taken and documents maintained to relevant checklists. The heads of the FAOs concerned review any gaps identified in the process and reach agreement with the QMS on actions to be taken to increase compliance. The next QCR follows up on the results of previous QCRs to ensure continuous improvement. Based on individual QCRs, the SQMS prepares a consolidated QCR report, using a QMF template that aggregates all significant findings, especially those with systemic implications. This consolidated report is submitted to the auditor-general after every QCR. The SQMS is also required to produce an annual QCR report that documents strategic and operational constraints noted during different QCRs and indicates actions to address them. The annual QCR is also required to identify good FAO practices for wider diffusion.
  • Evaluation and Grading of Audit Reports: The QMF stipulates that quality control committees (QCC) review all audit reports for quality and defines the composition and modus operandi of the QCCs. One QCC is established for each of the six audit wings and is chaired by the head of that wing. According to the QMF design, the QCC of one audit wing reviews and grades audit reports of another wing. Thus, audit reports are evaluated and graded by people who were not involved in conducting the audits or drafting the related audit reports. Rating criteria include such factors as the report structure; coherence of audit findings in describing criteria, condition, cause, and effect and recommendations; treatment and depiction of the client’s response, and quality of documentation. [6] Based on this review, the QCCs will give “A,” “B,” or “C” ratings to audit reports. Officers of an FAO whose reports get an “A” rating are considered for extra rewards. FAOs are required to make changes in their reports at the direction of their respective QCCs.

Quality Improvement Mechanism: Like any effective benchmarking process, the QMF supports continuous improvements in audit quality. FAOs get QCR reports showing what needs to be done to promote compliance with prescribed standards and are required to take actions agreed to with the findings of the QMSs, since the follow-up QCRs will invariably check and report on the implementation status of those actions. Audit wings get reports on quality aspects of audit operations from FAOs and the SQMS and are expected to design and implement preventive and corrective measures. Thus, continuous feedback in the form of periodic reports from internal and external quality mechanisms and timely implementation of corrective measures promote continuous improvement in audit quality. The AQMW closely monitors QMF implementation to identify opportunities to refine it.

Expect and Respond to Organizational Resistance to Change

Introducing and institutionalizing a QMF in the SAI of Pakistan was a gargantuan effort designed to bring the performance of audit wings and FAOs under closer organizational scrutiny. It was a radical change, as auditors were not used to their work being externally reviewed for quality. Thus, we anticipated some resistance to change and adopted a strategy to counter it. Our experience offers the following lessons that may be relevant to other SAIs interested in strengthening their quality management regimes.

  • Form a core team of officers with interest and experience in quality management: Developing a mechanism for quality management is likely to be new territory for most SAIs. However, each SAI is likely to have a few officers with interest and experience in this area. These officers should be identified and made into a core team that is given detailed orientation about what needs to be done. It would be worthwhile to involve experts from the marketplace. In designing its QMF, the SAI of Pakistan not only engaged such experts but also collaborated with the government Higher Education Commission, which had developed accreditation criteria and other benchmarks for higher education.
  • Involve stakeholders in understanding and designing quality management arrangements: When people participate in developing benchmarks against which their performance will be appraised, they are more likely to try to meet performance expectations. Auditors and supervisors may argue that applying the QMF will increase their work and jeopardize its quality. Thus, it is important to emphasize that even if there were no QMF, auditors and supervisors would still be required to meet prescribed standards. Workshops should be organized to help stakeholders understand the limitations of existing quality management systems and how a comprehensive QMF will improve their work quality. Workshop design should prompt participants to provide meaningful feedback that creates opportunities to refine the framework. Working intensively with its stakeholders, the SAI of Pakistan developed a shared understanding of the need to implement a quality management regime.
  • Establish organizational arrangements for quality management: Before implementing a QMF, an SAI must set up a dedicated and properly staffed organizational unit to support quality management within the SAI. Officers who will staff this new organizational unit—whether from inside the SAI or from outside sources—should be given the necessary orientation. The SAI of Pakistan has used a mix of its regular officers and experts from the marketplace to implement its QMF.
  • Ensure that audit offices clearly understand how their work will be evaluated: Audit offices are not used to being audited. The SAI of Pakistan’s participative approach softened up the ground for change. What remained was preparing FAOs to actually support the institutionalization of a new quality management regime. Workshops should be organized to sensitize officers to their roles and responsibilities under a new paradigm. The orientation process should continue for some years so that good practices of quality management are properly embedded in audit offices. The SAI Pakistan found that organizing such workshops for heads of audit wings/FAOs helped in institutionalizing our QMF.
  • Ensure consistency in QCRs: The quality of audit work may vary between audit offices located in different geographic locations. There may also be differences in the capabilities of staff working in quality management. To ensure that the QMF is applied consistently, staff need to be oriented to bring them onto the same page and result in consistent evaluation. The AQMW organizes such orientation sessions every time it starts a QCR.
  • Set the bar low in the beginning but gradually raise it: The QMF provides mechanisms that will gradually push audit offices into full compliance with requirements. Thus, in the early years, the SAIs should set a low bar for their audit offices and then gradually raise it. In the SAI of Pakistan, the QMSs’ grading of audit and exception reports is, by design, not as stringent as it could be. The AQMW is gradually increasing the rigor of its appraisal.
  • Integrate quality dimensions in capacity-building programs: Training interventions should emphasize the need for quality at every stage in the audit cycle to promote a culture of due diligence among auditors. We continue to work with our training establishment to mainstream quality dimensions in our training programs.

Conclusions

Aligning audit codes and manuals with International Standards on Auditing or ISSAIs is the first critical step in improving the quality of SAI work. However, significant progress in improving quality will occur only if auditors apply prescribed standards in their work, which is easier said than done. Our QMF demonstrates how an SAI may handle this challenge. It makes a clear distinction between quality assurance and quality control processes and shows how the two, working together, can increase compliance with prescribed auditing standards and enhance the effectiveness of an SAI’s audit operations. Introducing a QMF will trigger resistance to change with an SAI and the latter needs to take appropriate steps to mitigate this risk. In the SAI of Pakistan, the impact of implementing the QMF is already visible in improved compliance with prescribed standards and the quality of audit reports.

For additional information, contact the author at arsalanhaneef@yahoo.com.

 

[1] The term knowledge product refers to templates (i.e., audit planning and reporting templates) and guidance notes included therein to help auditors apply FAM good practices in their work. Some of products are available on the SAI Pakistan’s website http://www.agp.gov.pk.

[2] The QMF document is available on the SAI of Pakistan’s website: http://www.agp.gov.pk.

[3] ISSAI 40 requires that the quality control regime support this goal.

[4] Based on the FAM, the SAI of Pakistan has produced 24 such guidelines that facilitate the application of the FAM in a specific sector. For example, audits conducting information system (IS) audits will use the sectoral audit guidelines for IS audit.

[5] Since the introduction of the QMF, the AQMW has conducted QCRs of the planning and execution phases.

[6] The SAI Pakistan developed a Handbook of Quality Controls in Performance Auditing in November 2011. The QCCs will appraise performance audit reports against criteria that capture the significance of performance audit topics, the objectives and methodology used by audit teams, and the language and structure of performance audit reports as laid down in the handbook.