Teacher Guided Training: An Efficient Framework for Knowledge Transfer

TL;DR

Teacher-guided training (TGT) enables efficient knowledge transfer from large pretrained models to compact models, reducing data requirements and improving performance in limited data scenarios across image and text tasks.

Contribution

The paper introduces TGT, a novel framework that leverages pretrained models' representations to train compact models without extensive data, using data-domain exploration techniques.

Findings

TGT improves accuracy on image classification benchmarks.

TGT enhances performance in text classification and retrieval tasks.

Theoretical bounds support data-efficient generalization.

Abstract

The remarkable performance gains realized by large pretrained models, e.g., GPT-3, hinge on the massive amounts of data they are exposed to during training. Analogously, distilling such large models to compact models for efficient deployment also necessitates a large amount of (labeled or unlabeled) training data. In this paper, we propose the teacher-guided training (TGT) framework for training a high-quality compact model that leverages the knowledge acquired by pretrained generative models, while obviating the need to go through a large volume of data. TGT exploits the fact that the teacher has acquired a good representation of the underlying data domain, which typically corresponds to a much lower dimensional manifold than the input space. Furthermore, we can use the teacher to explore input space more efficiently through sampling or gradient-based methods; thus, making TGT especially attractive for limited data or long-tail settings. We formally capture this benefit of proposed data-domain exploration in our generalization bounds. We find that TGT can improve accuracy on several image classification benchmarks as well as a range of text classification and retrieval tasks.