Channel Exchanging Networks for Multimodal and Multitask Dense Image Prediction

TL;DR

This paper introduces Channel-Exchanging-Network (CEN), a novel, self-adaptive framework that enhances multimodal and multitask dense image prediction by adaptively exchanging channels based on learned importance, improving performance across various scenarios.

Contribution

The paper proposes a parameter-free, self-guided channel exchange mechanism applicable to multimodal and multitask dense image prediction, unifying these tasks within a single framework.

Findings

CEN outperforms state-of-the-art methods in semantic segmentation and image translation.

Extensive ablation studies confirm the effectiveness of each component.

CEN demonstrates versatility across multiple dense prediction scenarios.

Abstract

Multimodal fusion and multitask learning are two vital topics in machine learning. Despite the fruitful progress, existing methods for both problems are still brittle to the same challenge -- it remains dilemmatic to integrate the common information across modalities (resp. tasks) meanwhile preserving the specific patterns of each modality (resp. task). Besides, while they are actually closely related to each other, multimodal fusion and multitask learning are rarely explored within the same methodological framework before. In this paper, we propose Channel-Exchanging-Network (CEN) which is self-adaptive, parameter-free, and more importantly, applicable for multimodal and multitask dense image prediction. At its core, CEN adaptively exchanges channels between subnetworks of different modalities. Specifically, the channel exchanging process is self-guided by individual channel importance that is measured by the magnitude of Batch-Normalization (BN) scaling factor during training. For the application of dense image prediction, the validity of CEN is tested by four different scenarios: multimodal fusion, cycle multimodal fusion, multitask learning, and multimodal multitask learning. Extensive experiments on semantic segmentation via RGB-D data and image translation through multi-domain input verify the effectiveness of CEN compared to state-of-the-art methods. Detailed ablation studies have also been carried out, which demonstrate the advantage of each component we propose. Our code is available at https://github.com/yikaiw/CEN.