When More Data Doesn't Help: Limits of Adaptation in Multitask Learning

TL;DR

This paper investigates the fundamental statistical limits of multitask learning, demonstrating that increased data per task does not necessarily improve adaptation, and establishes stronger impossibility results beyond previous no-free-lunch bounds.

Contribution

It provides a new impossibility result showing that abundant data per task cannot overcome the inherent hardness of multitask learning without distributional information.

Findings

Stronger impossibility results for multitask learning adaptation.

Increased sample size per task does not guarantee improved performance.

Highlights the importance of distributional information for effective multitask learning.

Abstract

Multitask learning and related frameworks have achieved tremendous success in modern applications. In multitask learning problem, we are given a set of heterogeneous datasets collected from related source tasks and hope to enhance the performance above what we could hope to achieve by solving each of them individually. The recent work of arXiv:2006.15785 has showed that, without access to distributional information, no algorithm based on aggregating samples alone can guarantee optimal risk as long as the sample size per task is bounded. In this paper, we focus on understanding the statistical limits of multitask learning. We go beyond the no-free-lunch theorem in arXiv:2006.15785 by establishing a stronger impossibility result of adaptation that holds for arbitrarily large sample size per task. This improvement conveys an important message that the hardness of multitask learning cannot be overcame by having abundant data per task. We also discuss the notion of optimal adaptivity that may be of future interests.