Consistency Training with Virtual Adversarial Discrete Perturbation

TL;DR

This paper introduces a novel consistency training method using virtual adversarial discrete perturbations to improve semi-supervised text classification and robustness by strategically adding discrete noise that maximizes prediction divergence.

Contribution

It proposes a new augmentation technique with discrete noise that effectively pushes decision boundaries, outperforming existing methods in text classification and robustness tasks.

Findings

Outperforms baseline consistency training methods

Effective in semi-supervised text classification

Enhances model robustness against perturbations

Abstract

Consistency training regularizes a model by enforcing predictions of original and perturbed inputs to be similar. Previous studies have proposed various augmentation methods for the perturbation but are limited in that they are agnostic to the training model. Thus, the perturbed samples may not aid in regularization due to their ease of classification from the model. In this context, we propose an augmentation method of adding a discrete noise that would incur the highest divergence between predictions. This virtual adversarial discrete noise obtained by replacing a small portion of tokens while keeping original semantics as much as possible efficiently pushes a training model's decision boundary. Experimental results show that our proposed method outperforms other consistency training baselines with text editing, paraphrasing, or a continuous noise on semi-supervised text classification tasks and a robustness benchmark