Certifying Language Model Robustness with Fuzzed Randomized Smoothing: An Efficient Defense Against Backdoor Attacks

TL;DR

This paper introduces Fuzzed Randomized Smoothing (FRS), a novel method that certifies language model robustness against backdoor attacks by efficiently identifying and randomizing vulnerable textual segments without needing poisoned training data.

Contribution

FRS combines robustness certification with model parameter smoothing and proactive fuzzing, offering a more efficient and broader certified robustness radius against backdoor attacks in language models.

Findings

FRS outperforms existing methods in defense efficiency and robustness.

It achieves a broader certified robustness radius.

Experimental results validate FRS's effectiveness across datasets and attack strategies.

Abstract

The widespread deployment of pre-trained language models (PLMs) has exposed them to textual backdoor attacks, particularly those planted during the pre-training stage. These attacks pose significant risks to high-reliability applications, as they can stealthily affect multiple downstream tasks. While certifying robustness against such threats is crucial, existing defenses struggle with the high-dimensional, interdependent nature of textual data and the lack of access to original poisoned pre-training data. To address these challenges, we introduce \textbf{F}uzzed \textbf{R}andomized \textbf{S}moothing (\textbf{FRS}), a novel approach for efficiently certifying language model robustness against backdoor attacks. FRS integrates software robustness certification techniques with biphased model parameter smoothing, employing Monte Carlo tree search for proactive fuzzing to identify vulnerable textual segments within the Damerau-Levenshtein space. This allows for targeted and efficient text randomization, while eliminating the need for access to poisoned training data during model smoothing. Our theoretical analysis demonstrates that FRS achieves a broader certified robustness radius compared to existing methods. Extensive experiments across various datasets, model configurations, and attack strategies validate FRS's superiority in terms of defense efficiency, accuracy, and robustness.