Adapted Deep Embeddings: A Synthesis of Methods for $k$-Shot Inductive Transfer Learning

TL;DR

This paper compares and combines methods from transfer learning, deep metric learning, and few-shot learning, demonstrating that hybrid adapted-embedding approaches significantly improve performance in $k$-shot inductive transfer tasks.

Contribution

It systematically evaluates and unifies three transfer learning paradigms, introducing hybrid adapted-embedding methods that outperform existing approaches in various domains.

Findings

Deep embeddings outperform weight transfer for transfer learning.

Hybrid adapted-embedding methods reduce error by 34% on average.

Histogram loss is the most robust for learning embeddings.

Abstract

The focus in machine learning has branched beyond training classifiers on a single task to investigating how previously acquired knowledge in a source domain can be leveraged to facilitate learning in a related target domain, known as inductive transfer learning. Three active lines of research have independently explored transfer learning using neural networks. In weight transfer, a model trained on the source domain is used as an initialization point for a network to be trained on the target domain. In deep metric learning, the source domain is used to construct an embedding that captures class structure in both the source and target domains. In few-shot learning, the focus is on generalizing well in the target domain based on a limited number of labeled examples. We compare state-of-the-art methods from these three paradigms and also explore hybrid adapted-embedding methods that use limited target-domain data to fine tune embeddings constructed from source-domain data. We conduct a systematic comparison of methods in a variety of domains, varying the number of labeled instances available in the target domain ($k$), as well as the number of target-domain classes. We reach three principal conclusions: (1) Deep embeddings are far superior, compared to weight transfer, as a starting point for inter-domain transfer or model re-use (2) Our hybrid methods robustly outperform every few-shot learning and every deep metric learning method previously proposed, with a mean error reduction of 34% over state-of-the-art. (3) Among loss functions for discovering embeddings, the histogram loss (Ustinova & Lempitsky, 2016) is most robust. We hope our results will motivate a unification of research in weight transfer, deep metric learning, and few-shot learning.