Cross-Modal Manifold Learning for Cross-modal Retrieval

TL;DR

This paper introduces a scalable cross-modal manifold learning algorithm that preserves both global and local geometries during alignment, enhancing retrieval accuracy across different data modalities.

Contribution

The proposed method uniquely maintains global and local structures simultaneously in a joint manifold space using partial correspondences and affinity matrices, improving cross-modal retrieval.

Findings

Outperforms existing methods on multimodal datasets

Effective in classification and regression tasks

Enhances computer-assisted diagnosis systems

Abstract

This paper presents a new scalable algorithm for cross-modal similarity preserving retrieval in a learnt manifold space. Unlike existing approaches that compromise between preserving global and local geometries, the proposed technique respects both simultaneously during manifold alignment. The global topologies are maintained by recovering underlying mapping functions in the joint manifold space by deploying partially corresponding instances. The inter-, and intra-modality affinity matrices are then computed to reinforce original data skeleton using perturbed minimum spanning tree (pMST), and maximizing the affinity among similar cross-modal instances, respectively. The performance of proposed algorithm is evaluated upon two multimodal image datasets (coronary atherosclerosis histology and brain MRI) for two applications: classification, and regression. Our exhaustive validations and results demonstrate the superiority of our technique over comparative methods and its feasibility for improving computer-assisted diagnosis systems, where disease-specific complementary information shall be aggregated and interpreted across modalities to form the final decision.