A regression approach to the two-dataset problem

TL;DR

This paper introduces regression models to address the two-dataset problem, enabling analysis of data heterogeneity from different sources and improving the validity of conclusions drawn from combined datasets.

Contribution

It develops novel regression-based methods and prediction error metrics to evaluate data collection differences and heterogeneity between datasets from different populations.

Findings

Effective in distinguishing dataset differences

Applicable to real-world data from diverse sources

Enhances validity of combined data analysis

Abstract

This paper considers the two-dataset problem, where data are collected from two potentially different populations sharing common aspects. This problem arises when data are collected by two different types of researchers or from two different sources. We may reach invalid conclusions without using knowledge about the data collection process. To address this problem, this paper develops statistical regression models focusing on the difference in measurement and proposes two prediction errors that help to evaluate the underlying data collection process. As a consequence, it is possible to discuss the heterogeneity/similarity of the set of predictors in terms of prediction. Two real datasets are selected to illustrate our method.