Dissecting RGB-D Learning for Improved Multi-modal Fusion

TL;DR

This paper introduces an analytical framework to dissect RGB-D multi-modal learning, revealing feature discrepancies and cooperation rules, and proposes a simple fusion strategy that improves performance across tasks.

Contribution

The paper presents a novel dissection method for RGB-D models, providing insights into feature interactions and proposing an effective fusion strategy based on these insights.

Findings

Discrepancy in cross-modal features identified

Hybrid cooperation rule enhances inference

Proposed fusion strategy improves multiple tasks

Abstract

In the RGB-D vision community, extensive research has been focused on designing multi-modal learning strategies and fusion structures. However, the complementary and fusion mechanisms in RGB-D models remain a black box. In this paper, we present an analytical framework and a novel score to dissect the RGB-D vision community. Our approach involves measuring proposed semantic variance and feature similarity across modalities and levels, conducting visual and quantitative analyzes on multi-modal learning through comprehensive experiments. Specifically, we investigate the consistency and specialty of features across modalities, evolution rules within each modality, and the collaboration logic used when optimizing a RGB-D model. Our studies reveal/verify several important findings, such as the discrepancy in cross-modal features and the hybrid multi-modal cooperation rule, which highlights consistency and specialty simultaneously for complementary inference. We also showcase the versatility of the proposed RGB-D dissection method and introduce a straightforward fusion strategy based on our findings, which delivers significant enhancements across various tasks and even other multi-modal data.