Health-ORSC-Bench: A Benchmark for Measuring Over-Refusal and Safety Completion in Health Context

TL;DR

This paper introduces Health-ORSC-Bench, a comprehensive benchmark to evaluate healthcare language models on over-refusal and safe completion, revealing challenges in balancing safety and helpfulness across different model sizes and types.

Contribution

It presents the first large-scale benchmark for measuring over-refusal and safe completion in healthcare LLMs, with an automated pipeline and human validation, and evaluates 30 models revealing calibration challenges.

Findings

Larger models tend to be more safety-pessimistic and over-refuse benign prompts.

Safety-optimized models often refuse up to 80% of hard benign prompts.

Model size and family significantly influence safety and utility balance.

Abstract

Safety alignment in Large Language Models is critical for healthcare; however, reliance on binary refusal boundaries often results in \emph{over-refusal} of benign queries or \emph{unsafe compliance} with harmful ones. While existing benchmarks measure these extremes, they fail to evaluate Safe Completion: the model's ability to maximise helpfulness on dual-use or borderline queries by providing safe, high-level guidance without crossing into actionable harm. We introduce \textbf{Health-ORSC-Bench}, the first large-scale benchmark designed to systematically measure \textbf{Over-Refusal} and \textbf{Safe Completion} quality in healthcare. Comprising 31,920 benign boundary prompts across seven health categories (e.g., self-harm, medical misinformation), our framework uses an automated pipeline with human validation to test models at varying levels of intent ambiguity. We evaluate 30 state-of-the-art LLMs, including GPT-5 and Claude-4, revealing a significant tension: safety-optimised models frequently refuse up to 80\% of "Hard" benign prompts, while domain-specific models often sacrifice safety for utility. Our findings demonstrate that model family and size significantly influence calibration: larger frontier models (e.g., GPT-5, Llama-4) exhibit "safety-pessimism" and higher over-refusal than smaller or MoE-based counterparts (e.g., Qwen-3-Next), highlighting that current LLMs struggle to balance refusal and compliance. Health-ORSC-Bench provides a rigorous standard for calibrating the next generation of medical AI assistants toward nuanced, safe, and helpful completions. The code and data will be released upon acceptance. \textcolor{red}{Warning: Some contents may include toxic or undesired contents.}