Meta-Tsallis-Entropy Minimization: A New Self-Training Approach for Domain Adaptation on Text Classification

TL;DR

This paper introduces Meta-Tsallis Entropy Minimization (MTEM), a novel meta-learning based method for domain adaptation in text classification that optimizes adaptive entropy measures to improve pseudo-labeling and model transfer across domains.

Contribution

The paper proposes MTEM, a new meta-learning approach that optimizes Tsallis entropy for better domain adaptation in text classification, with an efficient approximation and sampling mechanism.

Findings

MTEM improves BERT's domain adaptation performance by an average of 4% on benchmark datasets.

Theoretical proof of convergence for the meta-learning algorithm in MTEM.

MTEM effectively reduces sensitivity to prediction errors during self-training.

Abstract

Text classification is a fundamental task for natural language processing, and adapting text classification models across domains has broad applications. Self-training generates pseudo-examples from the model's predictions and iteratively trains on the pseudo-examples, i.e., minimizes the loss on the source domain and the Gibbs entropy on the target domain. However, Gibbs entropy is sensitive to prediction errors, and thus, self-training tends to fail when the domain shift is large. In this paper, we propose Meta-Tsallis Entropy minimization (MTEM), which applies a meta-learning algorithm to optimize the instance adaptive Tsallis entropy on the target domain. To reduce the computation cost of MTEM, we propose an approximation technique to approximate the Second-order derivation involved in the meta-learning. To efficiently generate pseudo labels, we propose an annealing sampling mechanism for exploring the model's prediction probability. Theoretically, we prove the convergence of the meta-learning algorithm in MTEM and analyze the effectiveness of MTEM in achieving domain adaptation. Experimentally, MTEM improves the adaptation performance of BERT with an average of 4 percent on the benchmark dataset.