Sufficiency-principled Transfer Learning via Model Averaging

TL;DR

This paper introduces a transfer learning framework based on the sufficiency principle, using model averaging to improve performance across diverse similarity scenarios, with theoretical guarantees and empirical validation.

Contribution

It develops a unified model averaging approach for transfer learning that incorporates the sufficiency principle, addressing limitations of existing methods and providing theoretical and empirical support.

Findings

Achieves asymptotic optimality and normality in multi-source linear regression.

Demonstrates robustness to negative transfer and privacy concerns.

Shows superior performance over traditional methods in simulations and real data.

Abstract

When the transferable set is unknowable, transfering informative knowledge as much as possible\textemdash a principle we refer to as \emph{sufficiency}, becomes crucial for enhancing transfer learning effectiveness. However, existing transfer learning methods not only overlook the sufficiency principle, but also rely on restrictive single-similarity assumptions (\eg individual or combinatorial similarity), leading to suboptimal performance. To address these limitations, we propose a sufficiency-principled transfer learning framework via unified model averaging algorithms, accommodating both individual and combinatorial similarities. Theoretically, we establish the asymptotic/high-probability optimality, enhanced convergence rate and asymptotic normality for multi-source linear regression models with a diverging number of parameters, achieving sufficiency, robustness to negative transfer, privacy protection and feasible statistical inference. Extensive simulations and an empirical data analysis of Beijing housing rental data demonstrate the promising superiority of our framework over conventional alternatives.