Learnability and Competition in High-Dimensional Multi-Component ICA

TL;DR

This paper develops a mean-field theory for high-dimensional multi-component ICA, revealing phase transitions, learnability boundaries, and competition effects that influence convergence and component recovery.

Contribution

It introduces an asymptotically exact mean-field framework for multi-component online ICA, capturing coupling effects and phase structures in high dimensions.

Findings

Identifies decoupled and competition regimes in multi-component ICA.

Derives explicit learnability boundaries based on data moments and initialization.

Validates theoretical predictions with experiments on synthetic and hyperspectral data.

Abstract

Independent Component Analysis (ICA) is a foundational tool for unsupervised representation learning, yet its high-dimensional theory remains largely limited to single-component recovery. We develop an asymptotically exact mean-field theory for multi-component online ICA, capturing the coupling induced by simultaneous learning and orthogonalization. In the high-dimensional limit, the joint empirical distribution of learned estimates and ground-truth components converges to a deterministic process, yielding a closed ODE system for the overlap matrix between learned directions and true components. This characterization reveals a genuinely multi-component, initialization-driven phase structure: a decoupled regime, where estimates align with distinct components and evolve nearly independently, and a competition regime, where overlapping initializations induce orthogonality-driven conflicts, slow reorientation, and delayed convergence. Our steady-state analysis gives explicit learnability boundaries and competition conditions linking step size, data moments, and initialization. These conditions show that larger higher-order moments and competition shrink the stable learning-rate window, increase convergence times, and predict a staircase phenomenon in which the number of recoverable components changes discretely with the learning rate. Experiments on synthetic data and hyperspectral remote sensing data validate the predicted trajectories and phase behavior.