Is Multi-Distribution Learning as Easy as PAC Learning: Sharp Rates with Bounded Label Noise

TL;DR

This paper investigates the complexity of multi-distribution learning with bounded label noise, revealing inherent slow rates and fundamental barriers that distinguish it from single-task learning.

Contribution

It introduces a structured hypothesis-testing framework and demonstrates that multi-distribution learning incurs unavoidable statistical costs and slower rates compared to single-source learning.

Findings

Learning across multiple distributions has slow rates scaling with k/ε^2.

Certifying near-optimality under bounded noise is inherently costly in multi-source settings.

There is a fundamental statistical separation between random and Massart noise in multi-source learning.

Abstract

Towards understanding the statistical complexity of learning from heterogeneous sources, we study the problem of multi-distribution learning. Given $k$ data sources, the goal is to output a classifier for each source by exploiting shared structure to reduce sample complexity. We focus on the bounded label noise setting to determine whether the fast $1/\epsilon$ rates achievable in single-task learning extend to this regime with minimal dependence on $k$. Surprisingly, we show that this is not the case. We demonstrate that learning across $k$ distributions inherently incurs slow rates scaling with $k/\epsilon^2$, even under constant noise levels, unless each distribution is learned separately. A key technical contribution is a structured hypothesis-testing framework that captures the statistical cost of certifying near-optimality under bounded noise-a cost we show is unavoidable in the multi-distribution setting. Finally, we prove that when competing with the stronger benchmark of each distribution's optimal Bayes error, the sample complexity incurs a \textit{multiplicative} penalty in $k$. This establishes a \textit{statistical} separation between random classification noise and Massart noise, highlighting a fundamental barrier unique to learning from multiple sources.