High-dimensional Joint Sparsity Random Effects Model for Multi-task Learning

TL;DR

This paper introduces a novel convex relaxation approach for multi-task learning that models joint sparsity as a random effects model, leading to improved performance over traditional group Lasso methods.

Contribution

The paper proposes a two-step convex relaxation method based on sparse covariance coding and ridge regression with a sparse quadratic regularizer, enhancing joint sparsity modeling.

Findings

Outperforms group Lasso in experiments

Produces asymptotically optimal quadratic regularizer

Effective in high-dimensional multi-task learning

Abstract

Joint sparsity regularization in multi-task learning has attracted much attention in recent years. The traditional convex formulation employs the group Lasso relaxation to achieve joint sparsity across tasks. Although this approach leads to a simple convex formulation, it suffers from several issues due to the looseness of the relaxation. To remedy this problem, we view jointly sparse multi-task learning as a specialized random effects model, and derive a convex relaxation approach that involves two steps. The first step learns the covariance matrix of the coefficients using a convex formulation which we refer to as sparse covariance coding; the second step solves a ridge regression problem with a sparse quadratic regularizer based on the covariance matrix obtained in the first step. It is shown that this approach produces an asymptotically optimal quadratic regularizer in the multitask learning setting when the number of tasks approaches infinity. Experimental results demonstrate that the convex formulation obtained via the proposed model significantly outperforms group Lasso (and related multi-stage formulations