Towards Universal and Black-Box Query-Response Only Attack on LLMs with QROA

TL;DR

QROA introduces a black-box, query-only attack method on LLMs that efficiently finds adversarial suffixes to bypass safety measures, revealing significant security vulnerabilities.

Contribution

This paper presents QROA, a novel black-box attack framework that does not require internal model access or human-crafted templates, advancing adversarial attack techniques on LLMs.

Findings

Achieves over 80% attack success rate across multiple models.

Operates solely through query-response interactions without internal model info.

Identifies universal adversarial suffixes for broad applicability.

Abstract

The rapid adoption of Large Language Models (LLMs) has exposed critical security and ethical vulnerabilities, particularly their susceptibility to adversarial manipulations. This paper introduces QROA, a novel black-box jailbreak method designed to identify adversarial suffixes that can bypass LLM alignment safeguards when appended to a malicious instruction. Unlike existing suffix-based jailbreak approaches, QROA does not require access to the model's logit or any other internal information. It also eliminates reliance on human-crafted templates, operating solely through the standard query-response interface of LLMs. By framing the attack as an optimization bandit problem, QROA employs a surrogate model and token level optimization to efficiently explore suffix variations. Furthermore, we propose QROA-UNV, an extension that identifies universal adversarial suffixes for individual models, enabling one-query jailbreaks across a wide range of instructions. Testing on multiple models demonstrates Attack Success Rate (ASR) greater than 80\%. These findings highlight critical vulnerabilities, emphasize the need for advanced defenses, and contribute to the development of more robust safety evaluations for secure AI deployment. The code is made public on the following link: https://github.com/qroa/QROA