Near-optimal and Efficient First-Order Algorithm for Multi-Task Learning with Shared Linear Representation

TL;DR

This paper presents a first-order algorithm for multi-task learning with shared linear representations that converges rapidly and achieves near-optimal estimation error, improving upon existing likelihood-based methods.

Contribution

It introduces a novel, efficient first-order algorithm for multi-task learning with shared linear representations, with theoretical guarantees and improved error bounds.

Findings

Converges in rac{}( ilde{O}(1)) iterations.

Achieves near-optimal estimation error of rac{}(dk/(TN)).

Improves error bounds over existing likelihood-based methods by a factor of k.

Abstract

Multi-task learning (MTL) has emerged as a pivotal paradigm in machine learning by leveraging shared structures across multiple related tasks. Despite its empirical success, the development of likelihood-based efficiently solvable algorithms--even for shared linear representations--remains largely underdeveloped, primarily due to the non-convex structure intrinsic to matrix factorization. This paper introduces a first-order algorithm that jointly learns a shared representation and task-specific parameters, with guaranteed efficiency. Notably, it converges in $\widetilde{\mathcal{O}}(1)$ iterations and attains a \emph{near-optimal} estimation error of $\widetilde{\mathcal{O}}(dk/(TN))$, \emph{improving} over existing likelihood-based methods by a factor of $k$, where $d$, $k$, $T$, $N$ denote input dimension, representation dimension, task count, and samples per task, respectively. Our results justify that likelihood-based first-order methods can efficiently solve the MTL problem.