CTRL Your Shift: Clustered Transfer Residual Learning for Many Small Datasets

TL;DR

This paper presents CTRL, a meta-learning approach designed to improve predictive accuracy and source-specific differentiation in machine learning tasks involving many small, heterogeneous datasets with distributional shifts.

Contribution

The paper introduces CTRL, a novel meta-learning method that effectively clusters and transfers residuals across sources, enabling better accuracy and heterogeneity preservation in small, diverse datasets.

Findings

CTRL outperforms state-of-the-art benchmarks on large-scale datasets.

High-quality clusters can be learned efficiently without repeated refitting.

CTRL maintains source-level heterogeneity while improving overall accuracy.

Abstract

Machine learning (ML) tasks often utilize large-scale data that is drawn from several distinct sources, such as different locations, treatment arms, or groups. In such settings, practitioners often desire predictions that not only exhibit good overall accuracy, but also remain reliable within each source and preserve the differences that matter across sources. For instance, several asylum and refugee resettlement programs now use ML-based employment predictions to guide where newly arriving families are placed within a host country, which requires generating informative and differentiated predictions for many and often small source locations. However, this task is made challenging by several common characteristics of the data in these settings: the presence of numerous distinct data sources, distributional shifts between them, and substantial variation in sample sizes across sources. This paper introduces Clustered Transfer Residual Learning (CTRL), a meta-learning method that combines the strengths of cross-domain residual learning and adaptive pooling/clustering in order to simultaneously improve overall accuracy and preserve source-level heterogeneity. We establish new theory showing that high-quality clusters can be learned efficiently, bypassing the need for repeated model refitting over candidate subsets. We evaluate CTRL alongside other state-of-the-art benchmarks on 5 large-scale datasets. This includes a dataset from the national asylum program in Switzerland, where the algorithmic geographic assignment of asylum seekers is currently being piloted. CTRL consistently outperforms the benchmarks across several key metrics and when using a range of different base learners.