A High-Dimensional Statistical Method for Optimizing Transfer Quantities in Multi-Source Transfer Learning

TL;DR

This paper introduces a theoretical framework and an algorithm for optimizing the number of source samples used in multi-source transfer learning, improving training efficiency and accuracy.

Contribution

It provides a novel high-dimensional statistical method to determine optimal transfer sample quantities and develops an architecture-agnostic algorithm implementing these insights.

Findings

The proposed algorithm outperforms state-of-the-art methods in accuracy.

It enhances data efficiency in multi-source transfer learning.

Experimental results validate the theoretical framework.

Abstract

Multi-source transfer learning provides an effective solution to data scarcity in real-world supervised learning scenarios by leveraging multiple source tasks. In this field, existing works typically use all available samples from sources in training, which constrains their training efficiency and may lead to suboptimal results. To address this, we propose a theoretical framework that answers the question: what is the optimal quantity of source samples needed from each source task to jointly train the target model? Specifically, we introduce a generalization error measure based on K-L divergence, and minimize it based on high-dimensional statistical analysis to determine the optimal transfer quantity for each source task. Additionally, we develop an architecture-agnostic and data-efficient algorithm OTQMS to implement our theoretical results for target model training in multi-source transfer learning. Experimental studies on diverse architectures and two real-world benchmark datasets show that our proposed algorithm significantly outperforms state-of-the-art approaches in both accuracy and data efficiency. The code and supplementary materials are available in https://github.com/zqy0126/OTQMS.