A methodological analysis of prompt perturbations and their effect on attack success rates

TL;DR

This paper systematically analyzes how different LLM alignment methods influence the models' vulnerability to prompt attacks, revealing that small prompt changes can significantly alter attack success rates and highlighting limitations of current benchmark evaluations.

Contribution

It provides a statistical, systematic comparison of alignment methods' impact on attack susceptibility and emphasizes the importance of prompt variation in vulnerability assessment.

Findings

Small prompt modifications significantly affect attack success rates.

Different alignment methods show varying susceptibility to prompt attacks.

Current benchmarks may not fully capture model vulnerabilities.

Abstract

This work aims to investigate how different Large Language Models (LLMs) alignment methods affect the models' responses to prompt attacks. We selected open source models based on the most common alignment methods, namely, Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Reinforcement Learning with Human Feedback (RLHF). We conducted a systematic analysis using statistical methods to verify how sensitive the Attack Success Rate (ASR) is when we apply variations to prompts designed to elicit inappropriate content from LLMs. Our results show that even small prompt modifications can significantly change the Attack Success Rate (ASR) according to the statistical tests we run, making the models more or less susceptible to types of attack. Critically, our results demonstrate that running existing 'attack benchmarks' alone may not be sufficient to elicit all possible vulnerabilities of both models and alignment methods. This paper thus contributes to ongoing efforts on model attack evaluation by means of systematic and statistically-based analyses of the different alignment methods and how sensitive their ASR is to prompt variation.