Adversarial Robustness for Unified Multi-Modal Encoders via Efficient Calibration

TL;DR

This paper investigates the adversarial vulnerabilities of unified multi-modal encoders and proposes an efficient calibration method that significantly enhances robustness across various modalities without altering pretrained models.

Contribution

It is the first comprehensive study on adversarial robustness in unified multi-modal encoders and introduces a modality-specific calibration approach that improves robustness with minimal additional training.

Findings

Adversarial perturbations cause substantial performance drops across all modalities.

The proposed calibration improves robustness by up to 47.3% at epsilon=4/255.

The method maintains or enhances clean performance with less than 1% additional parameters.

Abstract

Recent unified multi-modal encoders align a wide range of modalities into a shared representation space, enabling diverse cross-modal tasks. Despite their impressive capabilities, the robustness of these models under adversarial perturbations remains underexplored, which is a critical concern for safety-sensitive applications. In this work, we present the first comprehensive study of adversarial vulnerability in unified multi-modal encoders. We find that even mild adversarial perturbations lead to substantial performance drops across all modalities. Non-visual inputs, such as audio and point clouds, are especially fragile, while visual inputs like images and videos also degrade significantly. To address this, we propose an efficient adversarial calibration framework that improves robustness across modalities without modifying pretrained encoders or semantic centers, ensuring compatibility with existing foundation models. Our method introduces modality-specific projection heads trained solely on adversarial examples, while keeping the backbone and embeddings frozen. We explore three training objectives: fixed-center cross-entropy, clean-to-adversarial L2 alignment, and clean-adversarial InfoNCE, and we introduce a regularization strategy to ensure modality-consistent alignment under attack. Experiments on six modalities and three Bind-style models show that our method improves adversarial robustness by up to 47.3 percent at epsilon = 4/255, while preserving or even improving clean zero-shot and retrieval performance with less than 1 percent trainable parameters.