SCAT: Robust Self-supervised Contrastive Learning via Adversarial Training for Text Classification

TL;DR

SCAT introduces a label-free adversarial training framework using contrastive learning to enhance the robustness of NLP models against textual adversarial attacks, applicable to both training from scratch and improving pre-trained models.

Contribution

The paper presents SCAT, a novel self-supervised adversarial training method that does not rely on labeled data, improving robustness of NLP models against adversarial attacks.

Findings

SCAT effectively trains robust models from scratch.

It significantly enhances robustness of pre-trained language models.

Combining SCAT with supervised adversarial training yields further robustness improvements.

Abstract

Despite their promising performance across various natural language processing (NLP) tasks, current NLP systems are vulnerable to textual adversarial attacks. To defend against these attacks, most existing methods apply adversarial training by incorporating adversarial examples. However, these methods have to rely on ground-truth labels to generate adversarial examples, rendering it impractical for large-scale model pre-training which is commonly used nowadays for NLP and many other tasks. In this paper, we propose a novel learning framework called SCAT (Self-supervised Contrastive Learning via Adversarial Training), which can learn robust representations without requiring labeled data. Specifically, SCAT modifies random augmentations of the data in a fully labelfree manner to generate adversarial examples. Adversarial training is achieved by minimizing the contrastive loss between the augmentations and their adversarial counterparts. We evaluate SCAT on two text classification datasets using two state-of-the-art attack schemes proposed recently. Our results show that SCAT can not only train robust language models from scratch, but it can also significantly improve the robustness of existing pre-trained language models. Moreover, to demonstrate its flexibility, we show that SCAT can also be combined with supervised adversarial training to further enhance model robustness.