SafeCoT: Improving VLM Safety with Minimal Reasoning

TL;DR

SafeCoT is a lightweight, interpretable framework that improves the safety and refusal behavior of vision-language models by using minimal rule-based reasoning, reducing overrefusal, and enhancing generalization in safety-critical scenarios.

Contribution

SafeCoT introduces a minimal supervision, rule-based chain-of-thought approach to enhance VLM safety and refusal capabilities without relying on extensive safety annotations.

Findings

Significantly reduces overrefusal in VLMs

Enhances safety-related reasoning with minimal supervision

Improves generalization across multiple benchmarks

Abstract

Ensuring safe and appropriate responses from vision-language models (VLMs) remains a critical challenge, particularly in high-risk or ambiguous scenarios. We introduce SafeCoT, a lightweight, interpretable framework that leverages rule-based chain-of-thought (CoT) supervision to improve refusal behavior in VLMs. Unlike prior methods that rely on large-scale safety annotations or complex modeling, SafeCoT uses minimal supervision to help models reason about safety risks and make context-aware refusals. Experiments across multiple benchmarks show that SafeCoT significantly reduces overrefusal and enhances generalization, even with limited training data. Our approach offers a scalable solution for aligning VLMs with safety-critical objectives.