Evolving Contextual Safety in Multi-Modal Large Language Models via Inference-Time Self-Reflective Memory

TL;DR

This paper introduces a new benchmark for evaluating contextual safety in multi-modal large language models and proposes EchoSafe, a training-free method that enhances safety behavior through self-reflective memory during inference.

Contribution

It presents MM-SafetyBench++, a benchmark for contextual safety evaluation, and EchoSafe, a novel inference-time framework that improves safety by leveraging self-reflective memory.

Findings

EchoSafe outperforms existing methods on safety benchmarks.

The benchmark effectively measures models' ability to adapt safety behavior.

Self-reflective memory enhances context-aware safety reasoning.

Abstract

Multi-modal Large Language Models (MLLMs) have achieved remarkable performance across a wide range of visual reasoning tasks, yet their vulnerability to safety risks remains a pressing concern. While prior research primarily focuses on jailbreak defenses that detect and refuse explicitly unsafe inputs, such approaches often overlook contextual safety, which requires models to distinguish subtle contextual differences between scenarios that may appear similar but diverge significantly in safety intent. In this work, we present MM-SafetyBench++, a carefully curated benchmark designed for contextual safety evaluation. Specifically, for each unsafe image-text pair, we construct a corresponding safe counterpart through minimal modifications that flip the user intent while preserving the underlying contextual meaning, enabling controlled evaluation of whether models can adapt their safety behaviors based on contextual understanding. Further, we introduce EchoSafe, a training-free framework that maintains a self-reflective memory bank to accumulate and retrieve safety insights from prior interactions. By integrating relevant past experiences into current prompts, EchoSafe enables context-aware reasoning and continual evolution of safety behavior during inference. Extensive experiments on various multi-modal safety benchmarks demonstrate that EchoSafe consistently achieves superior performance, establishing a strong baseline for advancing contextual safety in MLLMs. All benchmark data and code are available at https://echosafe-mllm.github.io.