On the Power of Source Screening for Learning Shared Feature Extractors

TL;DR

This paper investigates how selecting the most relevant data sources can improve the learning of shared feature extractors, demonstrating that careful source screening can achieve optimal representation even with less data.

Contribution

It introduces the concept of source screening for shared feature learning, providing algorithms and heuristics to identify informative sources, and proves their effectiveness both theoretically and empirically.

Findings

Source screening can lead to minimax optimal subspace estimation.

Training on a selected subset of sources can match full data performance.

Algorithms effectively identify informative sources in practice.

Abstract

Learning with shared representation is widely recognized as an effective way to separate commonalities from heterogeneity across various heterogeneous sources. Most existing work includes all related data sources via simultaneously training a common feature extractor and source-specific heads. It is well understood that data sources with low relevance or poor quality may hinder representation learning. In this paper, we further dive into the question of which data sources should be learned jointly by focusing on the traditionally deemed ``good'' collection of sources, in which individual sources have similar relevance and qualities with respect to the true underlying common structure. Towards tractability, we focus on the linear setting where sources share a low-dimensional subspace. We find that source screening can play a central role in statistically optimal subspace estimation. We show that, for a broad class of problem instances, training on a carefully selected subset of sources suffices to achieve minimax optimality, even when a substantial portion of data is discarded. We formalize the notion of an informative subpopulation, develop algorithms and practical heuristics for identifying such subsets, and validate their effectiveness through both theoretical analysis and empirical evaluations on synthetic and real-world datasets.