Robust Multi-modal Task-oriented Communications with Redundancy-aware Representations

TL;DR

This paper introduces a robust multi-modal communication framework that efficiently compresses and transmits task-relevant information by minimizing redundancy and enhancing reliability over noisy channels.

Contribution

It proposes a novel two-stage variational information bottleneck approach with mutual information minimization to optimize multi-modal data compression and robustness.

Findings

Significantly improves emotion recognition accuracy under noisy conditions.

Reduces inter-modal redundancy while maintaining task performance.

Enhances robustness of multi-modal communication against channel distortion.

Abstract

Semantic communications for multi-modal data can transmit task-relevant information efficiently over noisy and bandwidth-limited channels. However, a key challenge is to simultaneously compress inter-modal redundancy and improve semantic reliability under channel distortion. To address the challenge, we propose a robust and efficient multi-modal task-oriented communication framework that integrates a two-stage variational information bottleneck (VIB) with mutual information (MI) redundancy minimization. In the first stage, we apply uni-modal VIB to compress each modality separately, i.e., text, audio, and video, while preserving task-specific features. To enhance efficiency, an MI minimization module with adversarial training is then used to suppress cross-modal dependencies and to promote complementarity rather than redundancy. In the second stage, a multi-modal VIB is further used to compress the fused representation and to enhance robustness against channel distortion. Experimental results on multi-modal emotion recognition tasks demonstrate that the proposed framework significantly outperforms existing baselines in accuracy and reliability, particularly under low signal-to-noise ratio regimes. Our work provides a principled framework that jointly optimizes modality-specific compression, inter-modal redundancy, and communication reliability.