Rethinking Multi-Modal Learning from Gradient Uncertainty

TL;DR

This paper introduces a Bayesian gradient calibration method for multi-modal learning that models gradient uncertainty to improve optimization, addressing persistent performance fluctuations.

Contribution

It proposes a novel Bayesian-oriented approach that explicitly models gradient uncertainty and adaptively weights gradients, enhancing multi-modal learning optimization.

Findings

Improved predictive accuracy across multiple datasets

Effective handling of gradient reliability fluctuations

Enhanced robustness in multi-modal learning scenarios

Abstract

Multi-Modal Learning (MML) integrates information from diverse modalities to improve predictive accuracy. While existing optimization strategies have made significant strides by mitigating gradient direction conflicts, we revisit MML from a gradient-based perspective to explore further improvements. Empirically, we observe an interesting phenomenon: performance fluctuations can persist in both conflict and non-conflict settings. Based on this, we argue that: beyond gradient direction, the intrinsic reliability of gradients acts as a decisive factor in optimization, necessitating the explicit modeling of gradient uncertainty. Guided by this insight, we propose Bayesian-Oriented Gradient Calibration for MML (BOGC-MML). Our approach explicitly models gradients as probability distributions to capture uncertainty, interpreting their precision as evidence within the framework of subjective logic and evidence theory. By subsequently aggregating these signals using a reduced Dempster's combination rule, BOGC-MML adaptively weights gradients based on their reliability to generate a calibrated update. Extensive experiments demonstrate the effectiveness and advantages of the proposed method.