Virtual Data Augmentation: A Robust and General Framework for Fine-tuning Pre-trained Models

TL;DR

This paper introduces Virtual Data Augmentation (VDA), a versatile framework that enhances the robustness of pre-trained language models by generating semantically relevant and diverse virtual data embeddings through a mixture model and regularized training.

Contribution

The paper proposes a novel VDA framework combining masked language models and Gaussian noise for effective data augmentation during PLM fine-tuning.

Findings

Improves robustness of PLMs against adversarial attacks

Reduces performance degradation under adversarial conditions

Demonstrates effectiveness across six datasets

Abstract

Recent works have shown that powerful pre-trained language models (PLM) can be fooled by small perturbations or intentional attacks. To solve this issue, various data augmentation techniques are proposed to improve the robustness of PLMs. However, it is still challenging to augment semantically relevant examples with sufficient diversity. In this work, we present Virtual Data Augmentation (VDA), a general framework for robustly fine-tuning PLMs. Based on the original token embeddings, we construct a multinomial mixture for augmenting virtual data embeddings, where a masked language model guarantees the semantic relevance and the Gaussian noise provides the augmentation diversity. Furthermore, a regularized training strategy is proposed to balance the two aspects. Extensive experiments on six datasets show that our approach is able to improve the robustness of PLMs and alleviate the performance degradation under adversarial attacks. Our codes and data are publicly available at \textcolor{blue}{\url{https://github.com/RUCAIBox/VDA}}.