Technical Articles



International Journal of Government Auditing – July 2005

Quality Assurance of Third-party Information

Since the 1990s, the demand for information on the societal effects of government policy has increased. Politicians and the public no longer take for granted that the government needs to be accountable only in terms of money and associated business processes. As a result, in recent years government bodies have increasingly tried to provide politicians and the public with information on the societal outcomes of government policy. Much of the policy information used for these purposes comes from outside sources, referred to as “third-party information.” It is very important that this information be of good quality. The need to ensure the reliability of third-party information has been neglected in every area of government, including rules and regulations as well as practice. This article discusses why it is so important to ensure the quality of policy information, with an emphasis on third-party information. It also sets forth an approach designed to ensure the reliability of this information by classifying it according to risk. This theoretical framework is illustrated with examples from the Dutch experience. Finally, the article relates the risk-based classification to measures to ensure reliability.

The Importance of Third-party Information and Quality Assurance

Good policy information is essential to gain a clear understanding of whether each ministry has achieved its policy goals, what it has done to achieve these goals, and what this has cost. However, it is not only public accountability that stands or falls with the quality of the information used: good quality information is also needed to establish internal measures and adjust policies appropriately in a timely manner. Much policy information takes the form of third-party information. In a recent Dutch survey, about three-quarters of the policy information studied came from outside sources.1 The proportion is likely to be substantial in other countries as well. Given the amount of third-party information used by the average government body and the importance of this information, ensuring data reliability is an important link in the whole quality assurance chain related to the policy process. Ensuring the quality of third-party information scarcely figures, however, in the statutory frameworks of many central and local governments. For example, the Dutch central government has a ministerial Order on Performance Data and Evaluations in Central Government, which provides the framework for ensuring the quality of nonfinancial policy information. Remarkably, the order does not explicitly mention third-party information; indeed, it omits the concept completely! Also, the attention paid to quality assurance of third-party information in practice is far from adequate. A recent survey by the Netherlands Court of Audit, for instance, revealed that no steps had been taken to ensure the quality of over a quarter of the third-party information under consideration.2

Categories of Third-party Information

Third-party information can come from a wide range of sources. Figure 1 shows just how wide-ranging these sources can be from the point of view of a Dutch ministry.

Figure 1: Sources of Third-party Information

Government bodies using information from third-party sources for budgeting, accounting, and/or policy purposes should ensure that it is reliable, irrespective of its origin. Given the amount of information and the multiplicity of sources involved, this is no easy task. The multiplicity of sources can, however, be structured by classifying them according to the reliability risk associated with the third-party information. Based on this risk criterion, we have identified four categories of sources:

  • organizations with a statutory duty to provide information,
  • organizations required to be accountable to the user of the information,
  • other organizations with an interest in the content of the information, and
  • other organizations without an interest in the content of the information.

Organizations with a Statutory Duty to Provide Information

The first category is information from sources that have a statutory duty to collect, process, and present information. Many countries have these types of organizations—for example, a central bureau of statistics. The risk associated with the quality of third-party information from sources in fulfillment of their statutory duty can be estimated as low, based on the presumption that the law already contains provisions to ensure the quality of the information.3

Accountable Organizations

The second category is information from sources that are required to account for the government money they have spent along with the information supplied. These include administering bodies, local governments (social security benefits), and organizations that receive government subsidies. We gauge the risk associated with information from this category as high, as these organizations have an interest in the information supplied.

Quality Risk Adequately Covered: The Case of Fishing and Mineral Consumption

The Ministry of Agriculture, Nature and Food Quality is sent data on the quantity of fish caught by fishing companies and on net amounts of minerals used in the farming industry. These data, which we classify as high risk, come from sources that are accountable to the ministry. The ministry is aware of this risk and tries to ensure quality by carrying out its own measurements of such things as quantities of shellfish in coastal waters. It also enters into annual working agreements with the General Inspectorate on the inspection of catches reported by fishermen at ports. The reports on amounts of minerals used in the farming industry are checked against administrative data.

Other Interested Sources

The remaining information suppliers can be divided into those with and without an interest in the content of the information. Organizations can have substantive or financial interests. The interest may be substantive if the information affects the public’s perception of the organization (e.g., passenger numbers in the case of a national airport). Or the interest may be financial in cases such as a commercial research agency, where there may be a link between the information supplied and possible future contracts. We gauge the risk associated with information from this category as medium-high.

Quality Risk Not Adequately Covered: The Case of Domestic Refuse

In order to monitor the effects of its waste policy, the Ministry of Housing, Planning and the Environment is sent statistics on collections of domestic wastes by the Waste Management Council. Some of the basic data on domestic refuse are supplied directly by waste processing companies or industry associations without being validated by independent outsiders. This information comes from interested information suppliers, with which we associate a medium-high risk. Additional measures would seem to be needed to ensure the quality of these data. The ministry has not taken any special steps to ensure that these risks are covered, e.g., by having the data validated by independent outsiders.

Other Sources without an Interest

The fourth category is information from sources that have collected the information on their own initiative and have no interest in how the government uses it. Examples from this category are some scientific university publications and monitoring information from supranational government bodies. It is not possible to give a blanket assessment of the risks associated with information from this category, as the quality of the information can differ from one supplier to another and one product to another. The risks will need to be assessed individually for each information flow. We gauge the risk associated with information from this category as medium-low.

Measures to Ensure Reliability

Under ideal conditions, if a government uses certain third-party information, it would know how reliable that information is. In many cases, however, the government does not have adequate power over the information source to guarantee reliability by intervening in the source’s production of the information. Nor is it desirable, for reasons of cost, for the government to check the reliability of all third-party information. In many cases, the government would not be in a position to ascertain reliability, as doing so would require access to the supplier’s research files at the very least; here again, the government has no power over the organization collecting the information.

We nevertheless believe that government bodies can do something to ensure the quality of third-party information. In some cases, they can check the information by carrying out research of their own—for example, trend analyses, probability tests based on comparisons with other research, and data verification based on comparisons with public sources. If the government body is the principal, it has a broader arsenal of measures it can take. Examples include the following:

  • requiring the organization carrying out the research to meet certain standards of professionalism;
  • setting up a steering committee or a group to act as a sounding board;
  • agreeing on the definitions of reliability, validity, and topicality to be used by the organization carrying out the research;
  • requiring that an auditor’s report be provided with the information supplied;
  • establishing administrative requirements for the information supplier to meet (e.g., the existence of administrative organization/quality manuals);
  • testing the contractor’s research methods; and
  • including in the contract a clause to the effect that an external audit of the research files may be conducted.

According to administrative organization theory, the way measures are used must be balanced with the perceived risk to reliability. This correlates with the classification of third-party information in the preceding section. If the information comes from an organization that has collected it in fulfillment of a statutory duty, there would seem to be no need for the government body to take further substantial measures to guarantee its reliability.

However, a government body can be expected to take (1) substantial measures when it uses information from organizations that are accountable to it and (2) some additional measures to guarantee the reliability of information from other interested organizations.

Figure 2 shows the relationship between the classification of third-party information and the use of measures to guarantee reliability.


Figure 2: Classification and Risk Profile of Third-party Information and Related Measures Needed to Guarantee Reliability

Conclusion

Society is increasingly demanding that government be transparent about the effects of its policies, and this is a trend that cannot be reversed. To meet this demand, governments need to provide the right policy information and ensure its quality. Since much of this information–referred to here as third-party information–comes from outside organizations, the government is often not in a position to check its quality because it has no powers over the sources of the information and because of the high costs of auditing. However, government bodies could do more than they do at present. We believe that a risk-based approach is a useful aid to deciding on appropriate measures to take. This approach is based on classifying information sources according to their risk level. Appropriate measures can then be taken based on the risk level. Using this approach, adequate certainty that the third-party information is reliable can be obtained at a relatively low cost.

For additional information, contact the authors at: f.israel@rekenkamer.nl and f.pot@rekenkamer.nl.

 


1 Staat van de beleidsinformatie 2004 [Policy information report 2004]. Lower House, 2003-2004 session, 29550, Nos. 1-2.

2 Lower House, 2003-2004 session, 29 550, Nos. 1-2, p. 35.

3 These information sources also engage in activities outside their statutory duty, performed under contract. These activities have the same risk profile as the “other interested sources” category.