Taking Advantage of Sparsity in Multi-Task Learning

TL;DR

This paper investigates the use of the Group Lasso for multi-task linear regression, demonstrating its ability to effectively select variables and provide strong theoretical guarantees under certain conditions, even as the number of tasks grows.

Contribution

It extends theoretical analysis of the Group Lasso in multi-task learning, showing improved bounds that are independent of the number of predictor variables.

Findings

Group Lasso achieves oracle inequalities and variable selection consistency.

Bounds on estimation error do not depend on the number of predictor variables in multi-task settings.

Results extend to general noise distributions with finite variance.

Abstract

We study the problem of estimating multiple linear regression equations for the purpose of both prediction and variable selection. Following recent work on multi-task learning Argyriou et al. [2008], we assume that the regression vectors share the same sparsity pattern. This means that the set of relevant predictor variables is the same across the different equations. This assumption leads us to consider the Group Lasso as a candidate estimation method. We show that this estimator enjoys nice sparsity oracle inequalities and variable selection properties. The results hold under a certain restricted eigenvalue condition and a coherence condition on the design matrix, which naturally extend recent work in Bickel et al. [2007], Lounici [2008]. In particular, in the multi-task learning scenario, in which the number of tasks can grow, we are able to remove completely the effect of the number of predictor variables in the bounds. Finally, we show how our results can be extended to more general noise distributions, of which we only require the variance to be finite.