Meta-learning of shared linear representations beyond well-specified linear regression

TL;DR

This paper extends meta-learning to general convex objectives, demonstrating conditions under which shared low-rank or clustered structures can be recovered, and proposing a polynomial-time algorithm for shared linear representation learning.

Contribution

It generalizes meta-learning beyond well-specified linear regression, providing theoretical guarantees for structure recovery under convex objectives and introducing an efficient algorithm.

Findings

Rank and clustered estimators recover structures with enough samples and tasks.

Subspace recovery is possible with a single sample per task if tasks scale exponentially.

Proposed polynomial-time algorithm effectively learns shared linear representations.

Abstract

Motivated by multi-task and meta-learning approaches, we consider the problem of learning structure shared by tasks or users, such as shared low-rank representations or clustered structures. While all previous works focus on well-specified linear regression, we consider more general convex objectives, where the structural low-rank and cluster assumptions are expressed on the optima of each function. We show that under mild assumptions such as \textit{Hessian concentration} and \textit{noise concentration at the optimum}, rank and clustered regularized estimators recover such structure, provided the number of samples per task and the number of tasks are large enough. We then study the problem of recovering the subspace in which all the solutions lie, in the setting where there is only a single sample per task: we show that in that case, the rank-constrained estimator can recover the subspace, but that the number of tasks needs to scale exponentially large with the dimension of the subspace. Finally, we provide a polynomial-time algorithm via nuclear norm constraints for learning a shared linear representation in the context of convex learning objectives.