Self-Regulated Data-Free Knowledge Amalgamation for Text Classification

TL;DR

This paper introduces STRATANET, a novel framework for data-free knowledge amalgamation that enables training compact text classifiers from multiple pre-trained teachers without access to original data.

Contribution

It proposes a new data-free approach combining a steerable data generator and a self-regulative amalgamation module for effective knowledge transfer.

Findings

STRATANET outperforms baseline models on benchmark datasets.

The method effectively amalgamates knowledge from multiple teachers.

It works under both data-driven and data-free scenarios.

Abstract

Recently, there has been a growing availability of pre-trained text models on various model repositories. These models greatly reduce the cost of training new models from scratch as they can be fine-tuned for specific tasks or trained on large datasets. However, these datasets may not be publicly accessible due to the privacy, security, or intellectual property issues. In this paper, we aim to develop a lightweight student network that can learn from multiple teacher models without accessing their original training data. Hence, we investigate Data-Free Knowledge Amalgamation (DFKA), a knowledge-transfer task that combines insights from multiple pre-trained teacher models and transfers them effectively to a compact student network. To accomplish this, we propose STRATANET, a modeling framework comprising: (a) a steerable data generator that produces text data tailored to each teacher and (b) an amalgamation module that implements a self-regulative strategy using confidence estimates from the teachers' different layers to selectively integrate their knowledge and train a versatile student. We evaluate our method on three benchmark text classification datasets with varying labels or domains. Empirically, we demonstrate that the student model learned using our STRATANET outperforms several baselines significantly under data-driven and data-free constraints.