ICLShield: Exploring and Mitigating In-Context Learning Backdoor Attacks

TL;DR

This paper introduces ICLShield, a novel defense mechanism against backdoor attacks in in-context learning for large language models, based on a dual-learning hypothesis and dynamic concept preference adjustment, achieving state-of-the-art results.

Contribution

It proposes the dual-learning hypothesis for understanding ICL backdoor vulnerabilities and introduces ICLShield, a dynamic defense method that effectively mitigates backdoor attacks in LLMs.

Findings

ICLShield significantly outperforms existing defenses (+26.02% accuracy)

The dual-learning hypothesis explains the dominance of backdoor concepts in ICL vulnerabilities

ICLShield maintains high effectiveness on closed-source models like GPT-4

Abstract

In-context learning (ICL) has demonstrated remarkable success in large language models (LLMs) due to its adaptability and parameter-free nature. However, it also introduces a critical vulnerability to backdoor attacks, where adversaries can manipulate LLM behaviors by simply poisoning a few ICL demonstrations. In this paper, we propose, for the first time, the dual-learning hypothesis, which posits that LLMs simultaneously learn both the task-relevant latent concepts and backdoor latent concepts within poisoned demonstrations, jointly influencing the probability of model outputs. Through theoretical analysis, we derive an upper bound for ICL backdoor effects, revealing that the vulnerability is dominated by the concept preference ratio between the task and the backdoor. Motivated by these findings, we propose ICLShield, a defense mechanism that dynamically adjusts the concept preference ratio. Our method encourages LLMs to select clean demonstrations during the ICL phase by leveraging confidence and similarity scores, effectively mitigating susceptibility to backdoor attacks. Extensive experiments across multiple LLMs and tasks demonstrate that our method achieves state-of-the-art defense effectiveness, significantly outperforming existing approaches (+26.02% on average). Furthermore, our method exhibits exceptional adaptability and defensive performance even for closed-source models (e.g., GPT-4).