Hierarchical Mutual Information Analysis: Towards Multi-view Clustering in The Wild

TL;DR

This paper introduces a hierarchical deep multi-view clustering framework that effectively handles missing and unaligned data by combining data recovery and alignment techniques to improve clustering performance in real-world scenarios.

Contribution

It presents a novel deep MVC approach that separately addresses missing views and unalignment using dual prediction and contrastive reconstruction, a first in the field.

Findings

Outperforms state-of-the-art methods on public datasets

Effectively handles view missing and unalignment issues

Significantly improves clustering accuracy in challenging conditions

Abstract

Multi-view clustering (MVC) can explore common semantics from unsupervised views generated by different sources, and thus has been extensively used in applications of practical computer vision. Due to the spatio-temporal asynchronism, multi-view data often suffer from view missing and are unaligned in real-world applications, which makes it difficult to learn consistent representations. To address the above issues, this work proposes a deep MVC framework where data recovery and alignment are fused in a hierarchically consistent way to maximize the mutual information among different views and ensure the consistency of their latent spaces. More specifically, we first leverage dual prediction to fill in missing views while achieving the instance-level alignment, and then take the contrastive reconstruction to achieve the class-level alignment. To the best of our knowledge, this could be the first successful attempt to handle the missing and unaligned data problem separately with different learning paradigms. Extensive experiments on public datasets demonstrate that our method significantly outperforms state-of-the-art methods on multi-view clustering even in the cases of view missing and unalignment.