UniGuard: Towards Universal Safety Guardrails for Jailbreak Attacks on Multimodal Large Language Models

TL;DR

UniGuard is a universal safety mechanism for multimodal large language models that effectively defends against jailbreak attacks while preserving their core understanding abilities.

Contribution

It introduces a novel multimodal guardrail that considers both unimodal and cross-modal harmful signals, enhancing safety across various models and attack strategies.

Findings

Effective in reducing harmful outputs across multiple models

Maintains vision-language understanding capabilities

Generalizes well to different attack types

Abstract

Multimodal large language models (MLLMs) have revolutionized vision-language understanding but remain vulnerable to multimodal jailbreak attacks, where adversarial inputs are meticulously crafted to elicit harmful or inappropriate responses. We propose UniGuard, a novel multimodal safety guardrail that jointly considers the unimodal and cross-modal harmful signals. UniGuard trains a multimodal guardrail to minimize the likelihood of generating harmful responses in a toxic corpus. The guardrail can be seamlessly applied to any input prompt during inference with minimal computational costs. Extensive experiments demonstrate the generalizability of UniGuard across multiple modalities, attack strategies, and multiple state-of-the-art MLLMs, including LLaVA, Gemini Pro, GPT-4o, MiniGPT-4, and InstructBLIP. Notably, this robust defense mechanism maintains the models' overall vision-language understanding capabilities.