FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoning

TL;DR

FalseReject is a new resource with 16,000 queries and structured responses designed to improve safety in LLMs by reducing over-refusal, using a graph-informed adversarial framework and explicit reasoning.

Contribution

It introduces FalseReject, a comprehensive dataset and framework for enhancing safety alignment in LLMs through structured reasoning and adversarial prompt generation.

Findings

Supervised finetuning with FalseReject reduces over-refusal in LLMs.

Benchmarking shows persistent over-refusal issues across 29 SOTA LLMs.

FalseReject maintains safety and language capabilities after fine-tuning.

Abstract

Safety alignment approaches in large language models (LLMs) often lead to the over-refusal of benign queries, significantly diminishing their utility in sensitive scenarios. To address this challenge, we introduce FalseReject, a comprehensive resource containing 16k seemingly toxic queries accompanied by structured responses across 44 safety-related categories. We propose a graph-informed adversarial multi-agent interaction framework to generate diverse and complex prompts, while structuring responses with explicit reasoning to aid models in accurately distinguishing safe from unsafe contexts. FalseReject includes training datasets tailored for both standard instruction-tuned models and reasoning-oriented models, as well as a human-annotated benchmark test set. Our extensive benchmarking on 29 state-of-the-art (SOTA) LLMs reveals persistent over-refusal challenges. Empirical results demonstrate that supervised finetuning with FalseReject substantially reduces unnecessary refusals without compromising overall safety or general language capabilities.