Diagnosing and Mitigating Modality Interference in Multimodal Large Language Models

TL;DR

This paper identifies the issue of modality interference in multimodal large language models and proposes a unified finetuning approach with data augmentation and regularization to improve robustness and generalization across various benchmarks.

Contribution

It introduces a novel finetuning framework combining heuristic and adversarial perturbations with output consistency regularization to mitigate modality interference.

Findings

Enhanced robustness in unimodal tasks

Improved generalization across benchmarks

Maintained or improved multimodal performance

Abstract

Multimodal Large Language Models demonstrate strong performance on multimodal benchmarks, yet often exhibit poor robustness when exposed to spurious modality interference, such as irrelevant text in vision understanding, or irrelevant visual content in question answering. At its core, modality interference refers to cases where spurious signals from non-essential modalities distort model decisions, which we systematically analyze through causal, perturbation-based diagnostic experiments. To address this problem, we propose a unified finetuning framework that combines heuristic and adversarial perturbation-based data augmentation with output-level consistency regularization between original and perturbed inputs. Extensive experiments across image-heavy, text-heavy, and multimodal benchmarks, spanning multiple MLLM architectures and model scales, demonstrate consistent improvements in unimodal robustness and generalization, while improving standard multimodal performance.