Transfer Learning in $\ell_1$ Regularized Regression: Hyperparameter Selection Strategy based on Sharp Asymptotic Analysis

TL;DR

This paper provides a sharp asymptotic analysis of hyperparameter selection in transfer learning for high-dimensional sparse regression, revealing that ignoring one transfer type minimally impacts performance, thus simplifying hyperparameter tuning.

Contribution

It introduces a novel asymptotic analysis of hyperparameter selection in transfer learning algorithms like Trans-Lasso, showing that certain hyperparameters can be ignored without affecting performance.

Findings

Ignoring one transfer type has little effect on generalization.

Hyperparameter selection can be simplified significantly.

Theoretical results are validated on real-world datasets.

Abstract

Transfer learning techniques aim to leverage information from multiple related datasets to enhance prediction quality against a target dataset. Such methods have been adopted in the context of high-dimensional sparse regression, and some Lasso-based algorithms have been invented: Trans-Lasso and Pretraining Lasso are such examples. These algorithms require the statistician to select hyperparameters that control the extent and type of information transfer from related datasets. However, selection strategies for these hyperparameters, as well as the impact of these choices on the algorithm's performance, have been largely unexplored. To address this, we conduct a thorough, precise study of the algorithm in a high-dimensional setting via an asymptotic analysis using the replica method. Our approach reveals a surprisingly simple behavior of the algorithm: Ignoring one of the two types of information transferred to the fine-tuning stage has little effect on generalization performance, implying that efforts for hyperparameter selection can be significantly reduced. Our theoretical findings are also empirically supported by applications on real-world and semi-artificial datasets using the IMDb and MNIST datasets, respectively.