Claudini: Autoresearch Discovers State-of-the-Art Adversarial Attack Algorithms for LLMs

TL;DR

This paper introduces Claudini, an autoresearch pipeline using LLMs to discover novel, highly effective adversarial attack algorithms for language models, outperforming existing methods in jailbreaking and prompt injection tasks.

Contribution

The paper presents a method for automated discovery of adversarial attacks that outperform existing algorithms, demonstrating the potential for LLMs to advance security research autonomously.

Findings

Discovered attack algorithms achieve up to 40% success rate on CBRN queries.

Attacks generalize across models, achieving 100% success on Meta-SecAlign-70B.

Automated attack discovery outperforms all 30+ existing methods.

Abstract

LLM agents like Claude Code can not only write code but also be used for autonomous AI research and engineering \citep{rank2026posttrainbench, novikov2025alphaevolve}. We show that an \emph{autoresearch}-style pipeline \citep{karpathy2026autoresearch} powered by Claude Code discovers novel white-box adversarial attack \textit{algorithms} that \textbf{significantly outperform all existing (30+) methods} in jailbreaking and prompt injection evaluations. Starting from existing attack implementations, such as GCG~\citep{zou2023universal}, the agent iterates to produce new algorithms achieving up to 40\% attack success rate on CBRN queries against GPT-OSS-Safeguard-20B, compared to $\leq$10\% for existing algorithms (\Cref{fig:teaser}, left). The discovered algorithms generalize: attacks optimized on surrogate models transfer directly to held-out models, achieving \textbf{100\% ASR against Meta-SecAlign-70B} \citep{chen2025secalign} versus 56\% for the best baseline (\Cref{fig:teaser}, middle). Extending the findings of~\cite{carlini2025autoadvexbench}, our results are an early demonstration that incremental safety and security research can be automated using LLM agents. White-box adversarial red-teaming is particularly well-suited for this: existing methods provide strong starting points, and the optimization objective yields dense, quantitative feedback. We release all discovered attacks alongside baseline implementations and evaluation code at https://github.com/romovpa/claudini.