DCER: Dual-Stage Compression and Energy-Based Reconstruction

TL;DR

DCER introduces a dual-stage compression and energy-based reconstruction framework that enhances robustness in multimodal fusion, effectively handling noisy inputs and missing modalities, achieving state-of-the-art results.

Contribution

The paper presents a novel unified framework combining frequency-based compression and energy-based reconstruction to improve multimodal robustness against noise and missing data.

Findings

State-of-the-art performance on CMU-MOSI, CMU-MOSEI, and CH-SIMS datasets.

Energy-based uncertainty correlates strongly with prediction errors.

Robustness pattern favors multimodal fusion even with high missing modalities.

Abstract

Multimodal fusion faces two robustness challenges: noisy inputs degrade representation quality, and missing modalities cause prediction failures. We propose DCER, a unified framework addressing both challenges through dual-stage compression and energy-based reconstruction. The compression stage operates at two levels: within-modality frequency transforms (wavelet for audio, DCT for video) remove noise while preserving task-relevant patterns, and cross-modality bottleneck tokens force genuine integration rather than modality-specific shortcuts. For missing modalities, energy-based reconstruction recovers representations via gradient descent on a learned energy function, with the final energy providing intrinsic uncertainty quantification (\r{ho} > 0.72 correlation with prediction error). Experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS demonstrate state-of-the-art performance across all benchmarks, with a U-shaped robustness pattern favoring multimodal fusion at both complete and high-missing conditions. The code will be available on Github.