Towards Fundamental Limits for Active Multi-distribution Learning

TL;DR

This paper advances the understanding of active multi-distribution learning by developing new algorithms and establishing optimal bounds on label complexity, addressing both realizable and agnostic scenarios with theoretical rigor.

Contribution

It introduces new algorithms for active multi-distribution learning and proves optimal upper and lower bounds on label complexity in various settings.

Findings

Proves an upper bound of ( heta_{ ext{max}}(d+k)\u2206rac{1}{\u03b5}) in the near-realizable setting.

Establishes an upper bound involving ( heta_{ ext{max}}(d+k)(\u2206rac{1}{\u03b5}+rac{ u^2}{\u03b5^2})+rac{k u}{\u03b5^2}) in the agnostic setting.

Shows the realizable setting bound is information-theoretically optimal.

Abstract

Multi-distribution learning extends agnostic Probably Approximately Correct (PAC) learning to the setting in which a family of $k$ distributions, $\{D_i\}_{i\in[k]}$, is considered and a classifier's performance is measured by its error under the worst distribution. This problem has attracted a lot of recent interests due to its applications in collaborative learning, fairness, and robustness. Despite a rather complete picture of sample complexity of passive multi-distribution learning, research on active multi-distribution learning remains scarce, with algorithms whose optimality remaining unknown. In this paper, we develop new algorithms for active multi-distribution learning and establish improved label complexity upper and lower bounds, in distribution-dependent and distribution-free settings. Specifically, in the near-realizable setting we prove an upper bound of $\widetilde{O}\Bigl(\theta_{\max}(d+k)\ln\frac{1}{\varepsilon}\Bigr)$ and $\widetilde{O}\Bigl(\theta_{\max}(d+k)\Bigl(\ln\frac{1}{\varepsilon}+\frac{\nu^2}{\varepsilon^2}\Bigr)+\frac{k\nu}{\varepsilon^2}\Bigr)$ in the realizable and agnostic settings respectively, where $\theta_{\max}$ is the maximum disagreement coefficient among the $k$ distributions, $d$ is the VC dimension of the hypothesis class, $\nu$ is the multi-distribution error of the best hypothesis, and $\varepsilon$ is the target excess error. Moreover, we show that the bound in the realizable setting is information-theoretically optimal and that the $k\nu/\varepsilon^2$ term in the agnostic setting is fundamental for proper learners. We also establish instance-dependent sample complexity bound for passive multidistribution learning that smoothly interpolates between realizable and agnostic regimes~\citep{blum2017collaborative,zhang2024optimal}, which may be of independent interest.