PASAI Collaborates with CAAF to Develop Performance Measurement Systems in Two SAIs

Author: Yves Genest, Vice-president, Research and Special Projects, Canadian Audit and Accountability Foundation

In 2022-23, the Pacific Association of Supreme Audit Institutions (PASAI) commissioned the Canadian Audit and Accountability Foundation (CAAF), to support the design and implementation of performance management systems for the Office of the National Auditor of the Federated States of Micronesia (FSM) and the Pohnpei Office of the Public Auditor.

This project aimed to develop a performance monitoring system (PMS) to enable the two SAIs to monitor and report on their performance against their strategic plans. The PMS should enable each SAI to track the implementation of its strategic plan while also capturing the operational level key performance indicators (KPIs) and establish targets, baselines and milestones for key activities linked to the achievement of outputs and outcomes. 

A key element of the project was the conduct of very intense and interactive workshops with the Public Auditors of FSM and Pohnpei State and their respective staff. The outputs were PMS and monitoring plans containing several existing, new and potential key performance indicators, complete with methodology and procedures to collect them that will enable the SAIs to monitor their performance.

PASAI and CAAF identified the following key lessons learned resulting from the project.

• Design and implement a robust multi-year strategic plan

The central tenet of the INTOSAI-IDI Strategic Management Handbook for SAIs is the importance of conceptual and operational consistency between the SAI strategic framework and the result framework.  All the oversight and guiding documents of the SAI should use a consistent language both internally and externally and form the basis of the PMS. As the National Public Auditor of FSM, Haser Hainrick, said at the PMS workshop: We are already strong. We want to continue to improve by building on what has been accomplished.

• Start modestly and improve gradually

The SAIs had done considerable work to establish and conceptualize performance indicators. They were adamant to identify indicators that were cost-effective, practical and relevant to its goals and objectives. Throughout the workshops and working sessions many potential expansion and eventual improvements of the indicators that were discussed and adopted were considered in light of this principle.

• Leverage existing administrative databases

Basic indicators such as number of reports, hours of training provided, etc. are already collected for operational reasons. They presented low-hanging fruits for the SAIs to use. The SAIs proved to be very adept at taking advantage of them.

• Use of surveys and questionnaires

Surveys provide an opportunity to assess perception of various stakeholders receiving the services of the SAIs. In the case of SAIs it is usually audited organizations, legislators and general public. Each question in an exit survey of auditees that was developed for the SAI could be turned into an indicator for professionalism, transparency, efficiency and effectiveness. Although sometimes subjective and imperfect, they are cost-effective and generate new and fresh data going beyond mere administrative outputs. As the SAIs will mature these indicators can be displaced by more objective measurements or kept as complementary evidence. 

• Use the results framework to develop new indicators as the SAI matures

The PMS developed by this project were based directly on the SAIs’ Strategic Plans and were grounded on the outcomes, strategies, and activities outlined in Results Framework developed for this purpose.

The Results Frameworks provided a chain of results progressing from basic activities to ultimate outcomes. They are, therefore, gradually more sophisticated and require an increasing level of proficiency to collect as the SAI staff considers the various steps of chain of results (activities, outputs, immediate outcomes, intermediate outcomes, ultimate outcomes).  The SAIs can use this tool to identify gaps and invest efforts into filling them to continue to mature and improve. Examples of the progression in quality of KPIs is provided in the table below:

Table 1 – Examples of KPIs, Broken Down by Stages of the Results Framework 

• Leadership is the keystone of success

Leadership is the essential ingredient for the successful implementation of a PMS. The leaders of the SAIs not only demonstrated their commitments by adopting a conceptually sound PMS but also assigned resources for the data collection and committed to transparency of results through internal and external reporting. The PMS can therefore become a valid gauge of progress, or an early-warning system indicating that corrective actions must be taken.

• Be a model for the government

A PMS functions not only as a management tool but also as a mechanism of accountability. By using its PMS, linking it to a strategic framework, measuring its results and reporting them, the SAIs have developed a “best-in-class” approach to governance and accountability and can use the expertise they developed in their own organization to demonstrate the usefulness of this approach and to develop the expertise to assess government departments’ efforts to move in this direction. As the Public Auditor of the State of Pohnpei, Ihlen Joseph said at the workshop on PMS: I want my Office to be a positive example. We can show departments concretely how it can be done. 

Measuring the SAIs’ progress and achievements at the output and outcome level is a critical supporting element of the SAI’s strategic results framework and a core part of the strategy formulation process. The PMS will allow the SAIs to continuously assess and compare whether operational realities still allow and support the realization of the strategic intent. This will help assess progress with the implementation of planned outputs and outcomes as specified in the results framework of the SAI strategy.