Identifiable Shared Component Analysis of Unpaired Multimodal Mixtures

TL;DR

This paper extends shared component analysis to unpaired multi-modal data, proposing a divergence minimization approach with milder conditions for identifiability, validated on synthetic and real data.

Contribution

It introduces a novel method for identifying shared components in unaligned multi-modal mixtures using distribution divergence minimization.

Findings

Proposed a divergence-based loss for unpaired multi-modal data.

Derived sufficient conditions for shared component identifiability.

Validated results on synthetic and real-world datasets.

Abstract

A core task in multi-modal learning is to integrate information from multiple feature spaces (e.g., text and audio), offering modality-invariant essential representations of data. Recent research showed that, classical tools such as {\it canonical correlation analysis} (CCA) provably identify the shared components up to minor ambiguities, when samples in each modality are generated from a linear mixture of shared and private components. Such identifiability results were obtained under the condition that the cross-modality samples are aligned/paired according to their shared information. This work takes a step further, investigating shared component identifiability from multi-modal linear mixtures where cross-modality samples are unaligned. A distribution divergence minimization-based loss is proposed, under which a suite of sufficient conditions ensuring identifiability of the shared components are derived. Our conditions are based on cross-modality distribution discrepancy characterization and density-preserving transform removal, which are much milder than existing studies relying on independent component analysis. More relaxed conditions are also provided via adding reasonable structural constraints, motivated by available side information in various applications. The identifiability claims are thoroughly validated using synthetic and real-world data.