Multi-task Linear Regression without Eigenvalue Lower Bounds: Adaptivity, Robustness and Safety

TL;DR

This paper introduces a new estimator for multi-task linear regression that works under weaker spectral assumptions, ensuring adaptivity, robustness, and safety in high-dimensional, contaminated task settings.

Contribution

It proposes a matrix-weighted norm regularization approach and a balancedness condition that relaxes eigenvalue bounds, achieving near-minimax optimal prediction error.

Findings

Matches the rate of prior work under weaker assumptions

Ensures safety by performing no worse than independent learning in adverse cases

Achieves minimax optimality up to logarithmic factors in favorable regimes

Abstract

We study the multi-task linear regression problem in the presence of contaminated tasks. We address the setting where the unknown parameters of a majority of tasks are close in the $\ell_2$-norm, while a fraction of tasks are arbitrary outliers. Existing theoretical frameworks for this problem rely heavily on the assumption that the empirical second moment of each task has a minimum eigenvalue bounded away from zero (order $\Omega(1)$). Crucially, this assumption fails in many high-dimensional scenarios, rendering prior guarantees vacuous. To overcome this limitation, we propose an estimator based on matrix-weighted norm regularization. We also introduce a relative balancedness condition, quantified by a balancedness constant, that compares each task's second moment with the average inlier geometry and relaxes the need for taskwise second-moment lower bounds. In favorable regimes with moderate balancedness, our prediction MSE bounds match the rate of Duan and Wang (2023) under substantially weaker spectral assumptions; the resulting task-overall MSE is minimax optimal up to logarithmic factors. Furthermore, we demonstrate that our estimator enjoys a safety guarantee: when the relevant balancedness constant is large or infinite, or when tasks are unrelated, the method performs no worse than independent task learning. Consequently, our methodology achieves simultaneous adaptivity to task similarity, robustness to outliers, and safety outside favorable transfer regimes.