Distributionally Robust Multimodal Machine Learning

TL;DR

This paper introduces a distributionally robust optimization framework for multimodal machine learning, providing theoretical guarantees and empirical improvements in robustness for high-stakes applications.

Contribution

It proposes a novel DRO framework that captures modality-aware effects, with theoretical analysis and empirical validation demonstrating enhanced robustness.

Findings

Improved robustness in simulation and real-world datasets.

Theoretical performance guarantees through bounds.

Extension to encoder-specific error propagation settings.

Abstract

We consider the problem of distributionally robust multimodal machine learning. Existing approaches often rely on merging modalities on the feature level (early fusion) or heuristic uncertainty modeling, which downplays modality-aware effects and provide limited insights. We propose a novel distributionally robust optimization (DRO) framework that aims to study both the theoretical and practical insights of multimodal machine learning. We first justify this setup and show the significance of this problem through complexity analysis. We then establish both generalization upper bounds and minimax lower bounds which provide performance guarantees. These results are further extended in settings where we consider encoder-specific error propogations. Empirically, we demonstrate that our approach improves robustness in both simulation settings and real-world datasets. Together, these findings provide a principled foundation for employing multimodal machine learning models in high-stakes applications where uncertainty is unavoidable.