Searching for an Effective Defender: Benchmarking Defense against Adversarial Word Substitution

TL;DR

This paper systematically compares various defense methods against adversarial word substitution attacks on neural NLP models, proposing an improved defense that achieves state-of-the-art robustness on benchmark datasets.

Contribution

It provides a comprehensive comparison of defense strategies and introduces a new method that significantly enhances robustness against adversarial attacks.

Findings

Proposed method improves accuracy on both clean and adversarial data.

Achieved highest robustness on AGNEWS and IMDB datasets.

Provides insights into neural classifier behavior under attack.

Abstract

Recent studies have shown that deep neural networks are vulnerable to intentionally crafted adversarial examples, and various methods have been proposed to defend against adversarial word-substitution attacks for neural NLP models. However, there is a lack of systematic study on comparing different defense approaches under the same attacking setting. In this paper, we seek to fill the gap of systematic studies through comprehensive researches on understanding the behavior of neural text classifiers trained by various defense methods under representative adversarial attacks. In addition, we propose an effective method to further improve the robustness of neural text classifiers against such attacks and achieved the highest accuracy on both clean and adversarial examples on AGNEWS and IMDB datasets by a significant margin.