Uncertainty-Resilient Multimodal Learning via Consistency-Guided Cross-Modal Transfer

TL;DR

This paper introduces a framework for uncertainty-resilient multimodal learning that leverages consistency-guided cross-modal transfer to improve robustness and stability in noisy, heterogeneous data environments.

Contribution

It proposes a novel approach using semantic consistency in a shared latent space to enhance robustness and data efficiency in multimodal learning systems.

Findings

Significantly improves model stability and robustness to noise.

Enhances semantic robustness and data efficiency.

Captures reliable cross-modal structures under challenging conditions.

Abstract

Multimodal learning systems often face substantial uncertainty due to noisy data, low-quality labels, and heterogeneous modality characteristics. These issues become especially critical in human-computer interaction settings, where data quality, semantic reliability, and annotation consistency vary across users and recording conditions. This thesis tackles these challenges by exploring uncertainty-resilient multimodal learning through consistency-guided cross-modal transfer. The central idea is to use cross-modal semantic consistency as a basis for robust representation learning. By projecting heterogeneous modalities into a shared latent space, the proposed framework mitigates modality gaps and uncovers structural relations that support uncertainty estimation and stable feature learning. Building on this foundation, the thesis investigates strategies to enhance semantic robustness, improve data efficiency, and reduce the impact of noise and imperfect supervision without relying on large, high-quality annotations. Experiments on multimodal affect-recognition benchmarks demonstrate that consistency-guided cross-modal transfer significantly improves model stability, discriminative ability, and robustness to noisy or incomplete supervision. Latent space analyses further show that the framework captures reliable cross-modal structure even under challenging conditions. Overall, this thesis offers a unified perspective on resilient multimodal learning by integrating uncertainty modeling, semantic alignment, and data-efficient supervision, providing practical insights for developing reliable and adaptive brain-computer interface systems.