Hierarchical Mutual Distillation for Multi-View Fusion: Learning from All Possible View Combinations

TL;DR

This paper introduces a hierarchical mutual distillation approach that leverages all view combinations in multi-view learning, improving prediction accuracy and consistency by effectively managing view uncertainties.

Contribution

The novel MV-UWMD method performs hierarchical mutual distillation across all view combinations with an uncertainty-weighted mechanism, enhancing multi-view learning.

Findings

Improves prediction accuracy over existing methods.

Enhances consistency across multiple view predictions.

Effectively exploits view-specific information while reducing uncertainty impact.

Abstract

Multi-view learning often faces challenges in effectively leveraging images captured from different angles and locations. This challenge is particularly pronounced when addressing inconsistencies and uncertainties between views. In this paper, we propose a novel Multi-View Uncertainty-Weighted Mutual Distillation (MV-UWMD) method. Our method enhances prediction consistency by performing hierarchical mutual distillation across all possible view combinations, including single-view, partial multi-view, and full multi-view predictions. This introduces an uncertainty-based weighting mechanism through mutual distillation, allowing effective exploitation of unique information from each view while mitigating the impact of uncertain predictions. We extend a CNN-Transformer hybrid architecture to facilitate robust feature learning and integration across multiple view combinations. We conducted extensive experiments using a large, unstructured dataset captured from diverse, non-fixed viewpoints. The results demonstrate that MV-UWMD improves prediction accuracy and consistency compared to existing multi-view learning approaches.