High Dimensional Data Enrichment: Interpretable, Fast, and Data-Efficient

TL;DR

This paper introduces a new estimator for high-dimensional multi-task learning that captures shared and individual parameters, offering theoretical guarantees, an efficient algorithm, and experimental validation.

Contribution

It provides the first comprehensive statistical and computational analysis of data sharing in high-dimensional multi-task learning, including sample complexity and convergence guarantees.

Findings

The estimator effectively recovers shared and individual parameters.

All pooled samples contribute to the recovery of the common parameter.

The proposed iterative algorithm converges geometrically.

Abstract

We consider the problem of multi-task learning in the high dimensional setting. In particular, we introduce an estimator and investigate its statistical and computational properties for the problem of multiple connected linear regressions known as Data Enrichment/Sharing. The between-tasks connections are captured by a cross-tasks \emph{common parameter}, which gets refined by per-task \emph{individual parameters}. Any convex function, e.g., norm, can characterize the structure of both common and individual parameters. We delineate the sample complexity of our estimator and provide a high probability non-asymptotic bound for estimation error of all parameters under a geometric condition. We show that the recovery of the common parameter benefits from \emph{all} of the pooled samples. We propose an iterative estimation algorithm with a geometric convergence rate and supplement our theoretical analysis with experiments on synthetic data. Overall, we present a first thorough statistical and computational analysis of inference in the data-sharing model.