RECAP: A Resource-Efficient Method for Adversarial Prompting in Large Language Models

TL;DR

This paper presents a resource-efficient adversarial prompting method for large language models that matches prompts to a database of pre-trained adversarial examples, reducing computational costs while maintaining effectiveness.

Contribution

It introduces a novel approach that eliminates retraining by leveraging a database of adversarial prompts, enabling scalable and practical security evaluation of LLMs.

Findings

The method achieves competitive attack success rates.

Effectiveness varies with prompt type and algorithm.

Significant reduction in computational cost.

Abstract

The deployment of large language models (LLMs) has raised security concerns due to their susceptibility to producing harmful or policy-violating outputs when exposed to adversarial prompts. While alignment and guardrails mitigate common misuse, they remain vulnerable to automated jailbreaking methods such as GCG, PEZ, and GBDA, which generate adversarial suffixes via training and gradient-based search. Although effective, these methods particularly GCG are computationally expensive, limiting their practicality for organisations with constrained resources. This paper introduces a resource-efficient adversarial prompting approach that eliminates the need for retraining by matching new prompts to a database of pre-trained adversarial prompts. A dataset of 1,000 prompts was classified into seven harm-related categories, and GCG, PEZ, and GBDA were evaluated on a Llama 3 8B model to identify the most effective attack method per category. Results reveal a correlation between prompt type and algorithm effectiveness. By retrieving semantically similar successful adversarial prompts, the proposed method achieves competitive attack success rates with significantly reduced computational cost. This work provides a practical framework for scalable red-teaming and security evaluation of aligned LLMs, including in settings where model internals are inaccessible.