Plug, Play, and Fortify: A Low-Cost Module for Robust Multimodal Image Understanding Models

TL;DR

This paper introduces a low-cost, plug-and-play module that dynamically balances modality contributions in multimodal models using frequency domain analysis, significantly improving robustness to missing modalities.

Contribution

The work presents a novel Frequency Ratio Metric and a Multimodal Weight Allocation Module that enhance multimodal learning robustness and can be integrated into various architectures.

Findings

MWAM improves performance across diverse tasks and modalities.

The method enhances robustness to missing modalities.

It boosts state-of-the-art models addressing modality absence.

Abstract

Missing modalities present a fundamental challenge in multimodal models, often causing catastrophic performance degradation. Our observations suggest that this fragility stems from an imbalanced learning process, where the model develops an implicit preference for certain modalities, leading to the under-optimization of others. We propose a simple yet efficient method to address this challenge. The central insight of our work is that the dominance relationship between modalities can be effectively discerned and quantified in the frequency domain. To leverage this principle, we first introduce a Frequency Ratio Metric (FRM) to quantify modality preference by analyzing features in the frequency domain. Guided by FRM, we then propose a Multimodal Weight Allocation Module, a plug-and-play component that dynamically re-balances the contribution of each branch during training, promoting a more holistic learning paradigm. Extensive experiments demonstrate that MWAM can be seamlessly integrated into diverse architectural backbones, such as those based on CNNs and ViTs. Furthermore, MWAM delivers consistent performance gains across a wide range of tasks and modality combinations. This advancement extends beyond merely optimizing the performance of the base model; it also manifests as further performance improvements to state-of-the-art methods addressing the missing modality problem.