Machine Learning for the Production of Official Statistics: Density Ratio Estimation using Biased Transaction Data for Japanese labor statistics

TL;DR

This paper introduces a machine learning approach using density ratio estimation to produce timely official labor statistics from biased transaction data, demonstrating its effectiveness with Japanese employment data.

Contribution

It presents a novel application of density ratio estimation to correct selection bias in transaction data for official statistics production.

Findings

Early release of labor market indicators is possible using biased data.

The method effectively adjusts for selection bias in non-survey data.

Timely statistics can be produced without waiting for traditional survey results.

Abstract

National statistical institutes are beginning to use non-traditional data sources to produce official statistics. These sources, originally collected for non-statistical purposes, include point-of-sales(POS) data and mobile phone global positioning system(GPS) data. Such data have the potential to significantly enhance the usefulness of official statistics. In the era of big data, many private companies are accumulating vast amounts of transaction data. Exploring how to leverage these data for official statistics is increasingly important. However, progress has been slower than expected, mainly because such data are not collected through sample-based survey methods and therefore exhibit substantial selection bias. If this bias can be properly addressed, these data could become a valuable resource for official statistics, substantially expanding their scope and improving the quality of decision-making, including economic policy. This paper demonstrates that even biased transaction data can be useful for producing official statistics for prompt release, by drawing on the concepts of density ratio estimation and supervised learning under covariate shift, both developed in the field of machine learning. As a case study, we show that preliminary statistics can be produced in a timely manner using biased data from a Japanese private employment agency. This approach enables the early release of a key labor market indicator that would otherwise be delayed by up to a year, thereby making it unavailable for timely decision-making.