Fast Rates by Transferring from Auxiliary Hypotheses

TL;DR

This paper demonstrates that transferring auxiliary hypotheses in ERM-based linear algorithms can achieve fast generalization rates of O(1/m) when source hypotheses are well-aligned, otherwise reverting to standard rates.

Contribution

The work provides new theoretical bounds showing fast rates for transfer learning with auxiliary hypotheses and introduces a novel Rademacher complexity bound for smooth loss classes.

Findings

Fast rates of O(1/m) are achievable with good source hypotheses.

When source hypotheses are misaligned, standard learning rates are recovered.

A new Rademacher complexity bound under weaker assumptions is established.

Abstract

In this work we consider the learning setting where, in addition to the training set, the learner receives a collection of auxiliary hypotheses originating from other tasks. We focus on a broad class of ERM-based linear algorithms that can be instantiated with any non-negative smooth loss function and any strongly convex regularizer. We establish generalization and excess risk bounds, showing that, if the algorithm is fed with a good combination of source hypotheses, generalization happens at the fast rate $\mathcal{O}(1/m)$ instead of the usual $\mathcal{O}(1/\sqrt{m})$. On the other hand, if the source hypotheses combination is a misfit for the target task, we recover the usual learning rate. As a byproduct of our study, we also prove a new bound on the Rademacher complexity of the smooth loss class under weaker assumptions compared to previous works.