Changing Data Sources in the Age of Machine Learning for Official Statistics

TL;DR

This paper discusses the risks and challenges of changing data sources in machine learning-driven official statistics, emphasizing the importance of robustness, monitoring, and ethical considerations to maintain data integrity.

Contribution

It provides a comprehensive overview of the causes, risks, and mitigation strategies for data source changes in machine learning for official statistics.

Findings

Identifies key causes of data source changes including technical, ownership, and ethical factors.

Highlights effects such as concept drift, bias, and loss of data validity.

Recommends robustness and monitoring measures to ensure data quality and reliability.

Abstract

Data science has become increasingly essential for the production of official statistics, as it enables the automated collection, processing, and analysis of large amounts of data. With such data science practices in place, it enables more timely, more insightful and more flexible reporting. However, the quality and integrity of data-science-driven statistics rely on the accuracy and reliability of the data sources and the machine learning techniques that support them. In particular, changes in data sources are inevitable to occur and pose significant risks that are crucial to address in the context of machine learning for official statistics. This paper gives an overview of the main risks, liabilities, and uncertainties associated with changing data sources in the context of machine learning for official statistics. We provide a checklist of the most prevalent origins and causes of changing data sources; not only on a technical level but also regarding ownership, ethics, regulation, and public perception. Next, we highlight the repercussions of changing data sources on statistical reporting. These include technical effects such as concept drift, bias, availability, validity, accuracy and completeness, but also the neutrality and potential discontinuation of the statistical offering. We offer a few important precautionary measures, such as enhancing robustness in both data sourcing and statistical techniques, and thorough monitoring. In doing so, machine learning-based official statistics can maintain integrity, reliability, consistency, and relevance in policy-making, decision-making, and public discourse.