Reimagining Safety Alignment with An Image

TL;DR

This paper introduces Magic Image, a visual prompt framework that improves safety alignment in multimodal large language models by optimizing image prompts, reducing over-refusal, and supporting multiple value systems without parameter tuning.

Contribution

The paper presents a novel optimization-driven visual prompt method that enhances safety alignment and flexibility in MLLMs without requiring costly parameter updates.

Findings

Improved safety-effectiveness balance across datasets

Reduced over-refusal in safety mechanisms

Maintained model performance

Abstract

Large language models (LLMs) excel in diverse applications but face dual challenges: generating harmful content under jailbreak attacks and over-refusal of benign queries due to rigid safety mechanisms. These issues are further complicated by the need to accommodate different value systems and precisely align with given safety preferences. Moreover, traditional methods like SFT and RLHF lack this capability due to their costly parameter tuning requirements and inability to support multiple value systems within a single model. These problems are more obvious in multimodal large language models (MLLMs), especially in terms of heightened over-refusal in cross-modal tasks and new security risks arising from expanded attack surfaces. We propose Magic Image, an optimization-driven visual prompt framework that enhances security while reducing over-refusal. By optimizing image prompts using harmful/benign samples, our method enables a single model to adapt to different value systems and better align with given safety preferences without parameter updates. Experiments demonstrate improved safety-effectiveness balance across diverse datasets while preserving model performance, offering a practical solution for deployable MLLM safety alignment.