CARES: Comprehensive Evaluation of Safety and Adversarial Robustness in Medical LLMs

TL;DR

CARES is a comprehensive benchmark designed to evaluate and improve the safety and robustness of medical language models against adversarial prompts and jailbreak attacks, with detailed scoring and mitigation strategies.

Contribution

We introduce CARES, a detailed benchmark with a new evaluation protocol and safety score for assessing medical LLM safety and robustness against adversarial prompts.

Findings

Many state-of-the-art LLMs are vulnerable to jailbreaks.

Models tend to over-refuse safe but atypical queries.

A lightweight classifier can mitigate jailbreak attempts.

Abstract

Large language models (LLMs) are increasingly deployed in medical contexts, raising critical concerns about safety, alignment, and susceptibility to adversarial manipulation. While prior benchmarks assess model refusal capabilities for harmful prompts, they often lack clinical specificity, graded harmfulness levels, and coverage of jailbreak-style attacks. We introduce CARES (Clinical Adversarial Robustness and Evaluation of Safety), a benchmark for evaluating LLM safety in healthcare. CARES includes over 18,000 prompts spanning eight medical safety principles, four harm levels, and four prompting styles: direct, indirect, obfuscated, and role-play, to simulate both malicious and benign use cases. We propose a three-way response evaluation protocol (Accept, Caution, Refuse) and a fine-grained Safety Score metric to assess model behavior. Our analysis reveals that many state-of-the-art LLMs remain vulnerable to jailbreaks that subtly rephrase harmful prompts, while also over-refusing safe but atypically phrased queries. Finally, we propose a mitigation strategy using a lightweight classifier to detect jailbreak attempts and steer models toward safer behavior via reminder-based conditioning. CARES provides a rigorous framework for testing and improving medical LLM safety under adversarial and ambiguous conditions.