Q-MLLM: Vector Quantization for Robust Multimodal Large Language Model Security

TL;DR

Q-MLLM introduces a vector quantization-based architecture that enhances the security of multimodal large language models by effectively defending against adversarial visual attacks while maintaining reasoning capabilities.

Contribution

It proposes a novel two-level vector quantization method to discretize visual representations, blocking attack pathways and improving robustness against adversarial threats.

Findings

Achieves 100% defense success rate against jailbreak attacks.

Maintains competitive utility performance with minimal inference overhead.

Establishes vector quantization as an effective defense for multimodal AI security.

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in cross-modal understanding, but remain vulnerable to adversarial attacks through visual inputs despite robust textual safety mechanisms. These vulnerabilities arise from two core weaknesses: the continuous nature of visual representations, which allows for gradient-based attacks, and the inadequate transfer of text-based safety mechanisms to visual content. We introduce Q-MLLM, a novel architecture that integrates two-level vector quantization to create a discrete bottleneck against adversarial attacks while preserving multimodal reasoning capabilities. By discretizing visual representations at both pixel-patch and semantic levels, Q-MLLM blocks attack pathways and bridges the cross-modal safety alignment gap. Our two-stage training methodology ensures robust learning while maintaining model utility. Experiments demonstrate that Q-MLLM achieves significantly better defense success rate against both jailbreak attacks and toxic image attacks than existing approaches. Notably, Q-MLLM achieves perfect defense success rate (100\%) against jailbreak attacks except in one arguable case, while maintaining competitive performance on multiple utility benchmarks with minimal inference overhead. This work establishes vector quantization as an effective defense mechanism for secure multimodal AI systems without requiring expensive safety-specific fine-tuning or detection overhead. Code is available at https://github.com/Amadeuszhao/QMLLM.