Generalized Product-of-Experts for Learning Multimodal Representations in Noisy Environments

TL;DR

This paper introduces a generalized product-of-experts approach for robust multimodal representation learning in noisy environments, dynamically weighting modalities to improve performance on challenging benchmarks.

Contribution

It proposes a novel method that trains separate networks for each modality to assess credibility and dynamically combines them, enhancing robustness in noisy multimodal data.

Findings

Achieved state-of-the-art results on 3D hand-pose estimation benchmark.

Outperformed previous methods in multimodal surgical video segmentation.

Demonstrated robustness to noisy data through extensive evaluations.

Abstract

A real-world application or setting involves interaction between different modalities (e.g., video, speech, text). In order to process the multimodal information automatically and use it for an end application, Multimodal Representation Learning (MRL) has emerged as an active area of research in recent times. MRL involves learning reliable and robust representations of information from heterogeneous sources and fusing them. However, in practice, the data acquired from different sources are typically noisy. In some extreme cases, a noise of large magnitude can completely alter the semantics of the data leading to inconsistencies in the parallel multimodal data. In this paper, we propose a novel method for multimodal representation learning in a noisy environment via the generalized product of experts technique. In the proposed method, we train a separate network for each modality to assess the credibility of information coming from that modality, and subsequently, the contribution from each modality is dynamically varied while estimating the joint distribution. We evaluate our method on two challenging benchmarks from two diverse domains: multimodal 3D hand-pose estimation and multimodal surgical video segmentation. We attain state-of-the-art performance on both benchmarks. Our extensive quantitative and qualitative evaluations show the advantages of our method compared to previous approaches.