Dictionary-Aligned Concept Control for Safeguarding Multimodal LLMs

TL;DR

This paper introduces DACO, a novel framework using a large concept dictionary and sparse autoencoders to enhance safety in multimodal large language models by providing fine-grained activation control.

Contribution

DACO leverages a curated 15,000 concept dictionary and sparse coding to enable precise safety-related activation steering in MLLMs, improving safety without sacrificing performance.

Findings

DACO improves safety benchmarks like MM-SafetyBench and JailBreakV.

It maintains the general capabilities of MLLMs while enhancing safety.

Experiments on models like QwenVL, LLaVA, and InternVL validate DACO's effectiveness.

Abstract

Multimodal Large Language Models (MLLMs) have been shown to be vulnerable to malicious queries that can elicit unsafe responses. Recent work uses prompt engineering, response classification, or finetuning to improve MLLM safety. Nevertheless, such approaches are often ineffective against evolving malicious patterns, may require rerunning the query, or demand heavy computational resources. Steering the activations of a frozen model at inference time has recently emerged as a flexible and effective solution. However, existing steering methods for MLLMs typically handle only a narrow set of safety-related concepts or struggle to adjust specific concepts without affecting others. To address these challenges, we introduce Dictionary-Aligned Concept Control (DACO), a framework that utilizes a curated concept dictionary and a Sparse Autoencoder (SAE) to provide granular control over MLLM activations. First, we curate a dictionary of 15,000 multimodal concepts by retrieving over 400,000 caption-image stimuli and summarizing their activations into concept directions. We name the dataset DACO-400K. Second, we show that the curated dictionary can be used to intervene activations via sparse coding. Third, we propose a new steering approach that uses our dictionary to initialize the training of an SAE and automatically annotate the semantics of the SAE atoms for safeguarding MLLMs. Experiments on multiple MLLMs (e.g., QwenVL, LLaVA, InternVL) across safety benchmarks (e.g., MM-SafetyBench, JailBreakV) show that DACO significantly improves MLLM safety while maintaining general-purpose capabilities.