Perceive and Calibrate: Analyzing and Enhancing Robustness of Medical Multi-Modal Large Language Models

TL;DR

This paper systematically analyzes the impact of various noise perturbations on medical multi-modal large language models and introduces a training-free calibration framework to enhance their robustness in clinical settings.

Contribution

It presents a novel, training-free Inherent-enhanced Multi-modal Calibration framework that improves robustness of medical MLLMs against real-world noise across visual and textual modalities.

Findings

Achieves state-of-the-art robustness performance across multiple modalities.

Constructed a comprehensive benchmark with 11 noise types on 2 datasets.

Demonstrates effectiveness of perception-and-calibration approach in clinical scenarios.

Abstract

Medical Multi-modal Large Language Models (MLLMs) have shown promising clinical performance. However, their sensitivity to real-world input perturbations, such as imaging artifacts and textual errors, critically undermines their clinical applicability. Systematic analysis of such noise impact on medical MLLMs remains largely unexplored. Furthermore, while several works have investigated the MLLMs' robustness in general domains, they primarily focus on text modality and rely on costly fine-tuning. They are inadequate to address the complex noise patterns and fulfill the strict safety standards in medicine. To bridge this gap, this work systematically analyzes the impact of various perturbations on medical MLLMs across both visual and textual modalities. Building on our findings, we introduce a training-free Inherent-enhanced Multi-modal Calibration (IMC) framework that leverages MLLMs' inherent denoising capabilities following the perceive-and-calibrate principle for cross-modal robustness enhancement. For the visual modality, we propose a Perturbation-aware Denoising Calibration (PDC) which leverages MLLMs' own vision encoder to identify noise patterns and perform prototype-guided feature calibration. For text denoising, we design a Self-instantiated Multi-agent System (SMS) that exploits the MLLMs' self-assessment capabilities to refine noisy text through a cooperative hierarchy of agents. We construct a benchmark containing 11 types of noise across both image and text modalities on 2 datasets. Experimental results demonstrate our method achieves the state-of-the-art performance across multiple modalities, showing potential to enhance MLLMs' robustness in real clinical scenarios.