Multimodal diffusion geometry by joint diagonalization of Laplacians

TL;DR

This paper introduces a method to extend diffusion geometry to multi-modal data by joint diagonalization of Laplacians, improving manifold learning, retrieval, and clustering tasks.

Contribution

It presents a novel joint diagonalization approach for Laplacians that generalizes diffusion geometry to multiple modalities, unifying and enhancing spectral clustering techniques.

Findings

Joint diffusion geometry better captures multi-modal data structure.

Method improves manifold learning, retrieval, and clustering performance.

Many existing multimodal spectral clustering methods are special cases.

Abstract

We construct an extension of diffusion geometry to multiple modalities through joint approximate diagonalization of Laplacian matrices. This naturally extends classical data analysis tools based on spectral geometry, such as diffusion maps and spectral clustering. We provide several synthetic and real examples of manifold learning, retrieval, and clustering demonstrating that the joint diffusion geometry frequently better captures the inherent structure of multi-modal data. We also show that many previous attempts to construct multimodal spectral clustering can be seen as particular cases of joint approximate diagonalization of the Laplacians.