Efficient Adversarial Training in LLMs with Continuous Attacks

TL;DR

This paper introduces a computationally efficient adversarial training method for large language models by performing attacks in continuous embedding space, significantly improving robustness against discrete attacks while maintaining utility.

Contribution

It proposes a novel fast adversarial training algorithm (C-AdvUL) and an adversarial IPO variant (C-AdvIPO) that operate in embedding space, reducing computational costs for LLM robustness training.

Findings

Enhanced robustness of LLMs against discrete attacks

Maintained utility of models after adversarial training

Scalable approach applicable to various LLM architectures

Abstract

Large language models (LLMs) are vulnerable to adversarial attacks that can bypass their safety guardrails. In many domains, adversarial training has proven to be one of the most promising methods to reliably improve robustness against such attacks. Yet, in the context of LLMs, current methods for adversarial training are hindered by the high computational costs required to perform discrete adversarial attacks at each training iteration. We address this problem by instead calculating adversarial attacks in the continuous embedding space of the LLM, which is orders of magnitudes more efficient. We propose a fast adversarial training algorithm (C-AdvUL) composed of two losses: the first makes the model robust on continuous embedding attacks computed on an adversarial behaviour dataset; the second ensures the usefulness of the final model by fine-tuning on utility data. Moreover, we introduce C-AdvIPO, an adversarial variant of IPO that does not require utility data for adversarially robust alignment. Our empirical evaluation on five models from different families (Gemma, Phi3, Mistral, Zephyr, Llama2) and at different scales (2B, 3.8B, 7B) shows that both algorithms substantially enhance LLM robustness against discrete attacks (GCG, AutoDAN, PAIR), while maintaining utility. Our results demonstrate that robustness to continuous perturbations can extrapolate to discrete threat models. Thereby, we present a path toward scalable adversarial training algorithms for robustly aligning LLMs.