SDGO: Self-Discrimination-Guided Optimization for Consistent Safety in Large Language Models

TL;DR

SDGO introduces a reinforcement learning framework that aligns a large language model's discrimination and generation abilities, significantly improving safety against jailbreaking attacks without needing extra annotated data.

Contribution

The paper presents SDGO, a novel self-guided optimization method that enhances LLM safety by leveraging its own discrimination capabilities during training.

Findings

SDGO improves safety against jailbreaking attacks.

It maintains helpfulness on general benchmarks.

Requires no additional annotated data during training.

Abstract

Large Language Models (LLMs) excel at various natural language processing tasks but remain vulnerable to jailbreaking attacks that induce harmful content generation. In this paper, we reveal a critical safety inconsistency: LLMs can more effectively identify harmful requests as discriminators than defend against them as generators. This insight inspires us to explore aligning the model's inherent discrimination and generation capabilities. To this end, we propose SDGO (Self-Discrimination-Guided Optimization), a reinforcement learning framework that leverages the model's own discrimination capabilities as a reward signal to enhance generation safety through iterative self-improvement. Our method does not require any additional annotated data or external models during the training phase. Extensive experiments demonstrate that SDGO significantly improves model safety compared to both prompt-based and training-based baselines while maintaining helpfulness on general benchmarks. By aligning LLMs' discrimination and generation capabilities, SDGO brings robust performance against out-of-distribution (OOD) jailbreaking attacks. This alignment achieves tighter coupling between these two capabilities, enabling the model's generation capability to be further enhanced with only a small amount of discriminative samples. Our code and datasets are available at https://github.com/NJUNLP/SDGO.