By Tiare Rivera, Supreme Audit Institution of Chile (CGR)
Introduction
Supreme Audit Institutions (SAIs) are the cornerstone of maintaining accountability, transparency and effectiveness in the public sector, particularly in government operations. However, as technology evolves at breakneck speed, it is imperative that SAIs embrace cutting-edge data technologies, such as Machine Learning (ML), to revolutionize their auditing processes. With ML, SAIs can enhance efficiency, accuracy and effectiveness, providing a more comprehensive, data-driven analysis of government operations, thus ensuring the highest standards of accountability and trust.
In this article, we propose a roadmap for an SAI to incorporate these advanced data technologies with a focus on ML, mentioning the main algorithms currently in use and highlighting some important concepts. Finally, we try to show the impact that the use of ML can have on the performance of SAIs.
Engaging SAIs in Advanced Data Technologies
Several steps need to be taken to guide a Supreme Audit Institution towards the integration and effective use of advanced data technologies such as machine learning:
Develop a clear strategy: A data- and technology-enabled SAI must have a clear strategy that outlines the organization’s goals and objectives. It should be developed with input from all stakeholders, including auditors, IT professionals, and subject matter experts.
Invest in technology: SAIs need to invest in the technology and tools necessary to collect, store, and analyze large amounts of data. This includes investment in data management systems, data analysis tools, and AI/ML technologies.
Build a skilled workforce: To effectively use data and technology in the audit process, SAIs need a skilled workforce trained in data analysis, technology, and audit methodologies. This includes training auditors in data analysis techniques and hiring IT professionals and data scientists to support the audit process.
Promote data-driven decision making: SAIs need to promote data-driven decision-making throughout the organization. This includes using data to inform audit planning and risk assessment, and incorporating data analytics into the audit process.
Develop partnerships: SAIs should develop partnerships with other organizations, such as government agencies, to access and share data and to collaborate on the development and use of technology in the audit process.
Invest in data: Data collected and stored by SAIs needs to be cleaned, scaled, and transformed to produce results after advanced analysis.
Build the algorithms: The models to be used need to be selected according to the type of information being sought, and the model needs to be trained and fine-tuned to produce better results. The model must be evaluated on test sets for later application to real data, and the final models must be monitored and maintained on a regular basis.
Continuous monitoring and improvement: SAIs need to continuously monitor and evaluate the use of technology and data in the audit process to identify areas for improvement and make necessary adjustments. This includes regularly reviewing and updating the organization’s technology and data strategy.
Basic characteristics of machine learning algorithms
Once the SAI is ready for the next step, there can be two basic approaches to applying machine learning, depending on the availability of suitable data and whether there is an underlying structure in it.
Supervised learning: These algorithms are used to classify and predict outcomes based on labeled training data. They can be used in SAIs to classify transactions as fraudulent or not, predict the likelihood of fraud in a given area, or classify vendors as high or low risk.
Unsupervised learning: These algorithms are used to identify patterns and structure in unlabeled data. They can be used in SAIs to identify patterns of financial mismanagement, detect suspicious transactions, or identify outliers in spending patterns.
Types of Machine Learning algorithms that can be used by SAIs
There are several types of machine learning algorithms that can be used in SAIs to improve the efficiency, accuracy, and effectiveness of the audit process. See diagram for a broad overview.
Some of the most commonly used ML algorithms and their potential applications in SAIs include:
Clustering: These algorithms group similar data points together. They can be used in SAIs to group similar datasets, such as spending by department, or to identify groups of similar government programs or projects, making it easier to compare and evaluate their performance.
Anomaly detection: These algorithms are used to detect data points that deviate significantly from the norm. They can be used in SAIs to detect budgetary irregularities or to prioritize audits based on areas where performance deviates from expected patterns.
Artificial neural networks: These algorithms are modelled on the structure of the human brain and can be used for a variety of tasks, including image and speech recognition and natural language processing. They can be used in SAIs to process and analyze large amounts of unstructured data, such as text and images, and extract insights.
Decision Trees: These algorithms are used to classify data points based on a set of decision rules. They can be used in SAIs to classify transactions as fraudulent or not, to classify vendors as high or low risk, or to predict the likelihood of fraud in a given area.
K-Nearest Neighbors: These algorithms are used in image and video recognition, stock analysis and handwriting recognition applications. It uses labeled data points to label other points. The methodology is to create a voting system of nearest neighbors. The “k” is the number of neighbors that it checks. Its advantages are simplicity of implementation and that it works well with noisy data. The main disadvantage is that they require a significant amount of computation, which can be expensive for large datasets.
Potential barriers
While ML tools exist for fraud detection and financial oversight, it is not always easy to implement them operationally, adopt new digital tools, or integrate them into audit institutions. There are several barriers to machine learning innovation in SAIs and government agencies.
For example, many SAIs may lack accurate or consistent data, which can negatively affect the performance of machine learning algorithms and limit their effectiveness. A lack of technical expertise may affect SAIs’ ability to implement and successfully use machine learning algorithms, requiring specialized training and support. In addition, SAIs may need to integrate these algorithms into their existing systems and processes, which can be challenging and require significant resources.
Many agencies face cultural and structural barriers to change. These include a reluctance to innovate, a preference for the status quo, a fear of failure, excessive silos where different departments handle different data and segments of key missions, and a lack of leaders and managers who are skilled at facilitating change. In many organizations, the barriers to change are not only technical, but also structural, operational, managerial, and cultural. Unless leaders are committed to building a culture of innovation, the adoption of new technologies will almost always fall short of the intended benefits.
Ethical considerations and accountability:
Machine learning models have the potential to perpetuate or even exacerbate societal biases. Therefore, it is important to consider the ethical implications of the model, be aware of its limitations, and take steps to mitigate any negative consequences.
All of these concerns are limited in an environment that relies on human judgment. As a result, the use of machine learning in auditing to combat corruption and fraud poses unique ethical, legal, and governance challenges.
One of the biggest challenges is figuring out how to translate broad ethical principles such as fairness, equity, privacy, transparency, accountability, and human safety into concrete applications. These principles sometimes conflict, and SAIs need to define their meaning in the context of the particular system and determine how to evaluate machine learning algorithms accordingly.
In addition, some models are complex and difficult to interpret. It is important to understand the model’s decision-making process and to have interpretable models to explain the reasoning behind the model’s decision.
Expected impact of SAIs’ use of machine learning
One of the key benefits of incorporating advanced data technologies into the audit process is the ability to quickly and accurately analyze large amounts of data and identify patterns and trends that may not be obvious to human auditors. This can help SAIs identify potential fraud or financial mismanagement and make more informed decisions about where to focus their audit efforts.
By automating certain tasks, such as data entry and analysis, auditors can focus on more complex and high-value tasks, such as interpreting audit findings and making recommendations for improvement.
The use of machine learning by Supreme Audit Institutions has the potential to enhance SAIs’ ability to communicate and engage with the public by providing interactive tools such as real-time dashboards and customizable reports. These expected impacts could lead to a more transparent and accountable public sector, contributing to better governance and trust in institutions.
