Mixed-Sample SGD: an End-to-end Analysis of Supervised Transfer Learning

TL;DR

This paper develops an adaptive SGD method for supervised transfer learning that balances source and target data sampling, ensuring statistical guarantees and convergence, even without prior knowledge of source quality.

Contribution

It introduces a novel mixed-sample SGD algorithm that adaptively combines source and target data, providing transfer guarantees and convergence analysis for convex prediction tasks.

Findings

Converges at a $1/\sqrt{T}$ rate for linear regression with square loss.

Achieves adaptive statistical performance based on source quality.

Supported by experiments on synthetic and real datasets.

Abstract

Theoretical works on supervised transfer learning (STL) -- where the learner has access to labeled samples from both source and target distributions -- have for the most part focused on statistical aspects of the problem, while efficient optimization has received less attention. We consider the problem of designing an SGD procedure for STL that alternates sampling between source and target data, while maintaining statistical transfer guarantees without prior knowledge of the quality of the source data. A main algorithmic difficulty is in understanding how to design such an adaptive sub-sampling mechanism at each SGD step, to automatically gain from the source when it is informative, or bias towards the target and avoid negative transfer when the source is less informative. We show that, such a mixed-sample SGD procedure is feasible for general prediction tasks with convex losses, rooted in tracking an abstract sequence of constrained convex programs that serve to maintain the desired transfer guarantees. We instantiate these results in the concrete setting of linear regression with square loss, and show that the procedure converges, with $1/\sqrt{T}$ rate, to a solution whose statistical performance on the target is adaptive to the a priori unknown quality of the source. Experiments with synthetic and real datasets support the theory.