Understanding Transfer Learning via Mean-field Analysis

TL;DR

This paper introduces a mean-field theoretical framework to analyze the generalization errors in transfer learning, providing insights into convergence rates and benefits for neural networks in the mean-field regime.

Contribution

It develops a novel differential calculus approach on probability measures to study transfer learning, establishing convergence conditions and demonstrating benefits for neural networks.

Findings

Transfer learning generalization errors can be characterized using mean-field analysis.

Convergence rates depend on regularity and integrability conditions.

Transfer learning shows benefits in neural networks within the mean-field regime.

Abstract

We propose a novel framework for exploring generalization errors of transfer learning through the lens of differential calculus on the space of probability measures. In particular, we consider two main transfer learning scenarios, $\alpha$-ERM and fine-tuning with the KL-regularized empirical risk minimization and establish generic conditions under which the generalization error and the population risk convergence rates for these scenarios are studied. Based on our theoretical results, we show the benefits of transfer learning with a one-hidden-layer neural network in the mean-field regime under some suitable integrability and regularity assumptions on the loss and activation functions.