ELBA-Bench: An Efficient Learning Backdoor Attacks Benchmark for Large Language Models

TL;DR

ELBA-Bench is a comprehensive benchmark framework for evaluating backdoor attacks on large language models, covering various attack methods, datasets, and models, and providing insights into attack effectiveness and robustness.

Contribution

The paper introduces ELBA-Bench, a unified and extensive benchmark for backdoor attacks on LLMs, including over 1300 experiments and a universal toolbox for standardized research.

Findings

PEFT attacks outperform non-fine-tuning methods in classification tasks.

Optimized triggers improve attack robustness and cross-dataset generalization.

Backdoor techniques using task-relevant prompts enhance attack success while maintaining model performance.

Abstract

Generative large language models are crucial in natural language processing, but they are vulnerable to backdoor attacks, where subtle triggers compromise their behavior. Although backdoor attacks against LLMs are constantly emerging, existing benchmarks remain limited in terms of sufficient coverage of attack, metric system integrity, backdoor attack alignment. And existing pre-trained backdoor attacks are idealized in practice due to resource access constraints. Therefore we establish $\textit{ELBA-Bench}$, a comprehensive and unified framework that allows attackers to inject backdoor through parameter efficient fine-tuning ($\textit{e.g.,}$ LoRA) or without fine-tuning techniques ($\textit{e.g.,}$ In-context-learning). $\textit{ELBA-Bench}$ provides over 1300 experiments encompassing the implementations of 12 attack methods, 18 datasets, and 12 LLMs. Extensive experiments provide new invaluable findings into the strengths and limitations of various attack strategies. For instance, PEFT attack consistently outperform without fine-tuning approaches in classification tasks while showing strong cross-dataset generalization with optimized triggers boosting robustness; Task-relevant backdoor optimization techniques or attack prompts along with clean and adversarial demonstrations can enhance backdoor attack success while preserving model performance on clean samples. Additionally, we introduce a universal toolbox designed for standardized backdoor attack research, with the goal of propelling further progress in this vital area.