Deep Multimodal Fusion by Channel Exchanging

TL;DR

This paper introduces Channel-Exchanging-Network (CEN), a novel parameter-free multimodal fusion framework that dynamically exchanges channels between modality-specific sub-networks based on channel importance, improving performance in tasks like semantic segmentation and image translation.

Contribution

The paper proposes a new dynamic, parameter-free multimodal fusion method that balances inter-modal fusion and intra-modal processing by channel exchanging guided by BN scaling factors.

Findings

CEN outperforms current state-of-the-art methods in semantic segmentation and image translation.

The channel exchanging process improves multimodal fusion effectiveness.

Ablation studies confirm the importance of each component in CEN.

Abstract

Deep multimodal fusion by using multiple sources of data for classification or regression has exhibited a clear advantage over the unimodal counterpart on various applications. Yet, current methods including aggregation-based and alignment-based fusion are still inadequate in balancing the trade-off between inter-modal fusion and intra-modal processing, incurring a bottleneck of performance improvement. To this end, this paper proposes Channel-Exchanging-Network (CEN), a parameter-free multimodal fusion framework that dynamically exchanges channels between sub-networks of different modalities. Specifically, the channel exchanging process is self-guided by individual channel importance that is measured by the magnitude of Batch-Normalization (BN) scaling factor during training. The validity of such exchanging process is also guaranteed by sharing convolutional filters yet keeping separate BN layers across modalities, which, as an add-on benefit, allows our multimodal architecture to be almost as compact as a unimodal network. Extensive experiments on semantic segmentation via RGB-D data and image translation through multi-domain input verify the effectiveness of our CEN compared to current state-of-the-art methods. Detailed ablation studies have also been carried out, which provably affirm the advantage of each component we propose. Our code is available at https://github.com/yikaiw/CEN.