Transfer Learning with Pre-trained Conditional Generative Models

TL;DR

This paper introduces a transfer learning approach that leverages pre-trained conditional generative models to transfer knowledge without requiring overlapping labels, source data access, or identical target architectures, outperforming traditional methods.

Contribution

The paper proposes a novel transfer learning method using deep generative models with pseudo pre-training and semi-supervised learning stages, removing common assumptions in transfer learning.

Findings

Outperforms scratch training baselines.

Outperforms knowledge distillation methods.

Effective without source data access or label overlap.

Abstract

Transfer learning is crucial in training deep neural networks on new target tasks. Current transfer learning methods always assume at least one of (i) source and target task label spaces overlap, (ii) source datasets are available, and (iii) target network architectures are consistent with source ones. However, holding these assumptions is difficult in practical settings because the target task rarely has the same labels as the source task, the source dataset access is restricted due to storage costs and privacy, and the target architecture is often specialized to each task. To transfer source knowledge without these assumptions, we propose a transfer learning method that uses deep generative models and is composed of the following two stages: pseudo pre-training (PP) and pseudo semi-supervised learning (P-SSL). PP trains a target architecture with an artificial dataset synthesized by using conditional source generative models. P-SSL applies SSL algorithms to labeled target data and unlabeled pseudo samples, which are generated by cascading the source classifier and generative models to condition them with target samples. Our experimental results indicate that our method can outperform the baselines of scratch training and knowledge distillation.