Self-Evaluation as a Defense Against Adversarial Attacks on LLMs

TL;DR

This paper presents a self-evaluation based defense mechanism for LLMs against adversarial attacks, which does not require model fine-tuning and effectively reduces attack success rates.

Contribution

It introduces a novel, fine-tuning-free self-evaluation method that enhances robustness of LLMs against adversarial attacks, outperforming existing defenses.

Findings

Significantly reduces attack success rates on open and closed-source LLMs.

More resilient to attacks than existing methods like Llama-Guard2.

Does not require model fine-tuning, lowering implementation costs.

Abstract

We introduce a defense against adversarial attacks on LLMs utilizing self-evaluation. Our method requires no model fine-tuning, instead using pre-trained models to evaluate the inputs and outputs of a generator model, significantly reducing the cost of implementation in comparison to other, finetuning-based methods. Our method can significantly reduce the attack success rate of attacks on both open and closed-source LLMs, beyond the reductions demonstrated by Llama-Guard2 and commonly used content moderation APIs. We present an analysis of the effectiveness of our method, including attempts to attack the evaluator in various settings, demonstrating that it is also more resilient to attacks than existing methods. Code and data will be made available at https://github.com/Linlt-leon/self-eval.