GRP-Obliteration: Unaligning LLMs With a Single Unlabeled Prompt

TL;DR

This paper introduces GRP-Obliteration, a novel method that effectively unaligns safety-aligned large models using a single unlabeled prompt, preserving utility and outperforming existing techniques across various models and modalities.

Contribution

GRP-Obliteration is the first approach to reliably unalign models with a single unlabeled prompt, extending unalignment to diffusion models and outperforming prior methods.

Findings

Single prompt effectively unaligns safety models

Preserves model utility after unalignment

Generalizes to image generation systems

Abstract

Safety alignment is only as robust as its weakest failure mode. Despite extensive work on safety post-training, it has been shown that models can be readily unaligned through post-deployment fine-tuning. However, these methods often require extensive data curation and degrade model utility. In this work, we extend the practical limits of unalignment by introducing GRP-Obliteration (GRP-Oblit), a method that uses Group Relative Policy Optimization (GRPO) to directly remove safety constraints from target models. We show that a single unlabeled prompt is sufficient to reliably unalign safety-aligned models while largely preserving their utility, and that GRP-Oblit achieves stronger unalignment on average than existing state-of-the-art techniques. Moreover, GRP-Oblit generalizes beyond language models and can also unalign diffusion-based image generation systems. We evaluate GRP-Oblit on six utility benchmarks and five safety benchmarks across fifteen 7-20B parameter models, spanning instruct and reasoning models, as well as dense and MoE architectures. The evaluated model families include GPT-OSS, distilled DeepSeek, Gemma, Llama, Ministral, and Qwen.