StablePCA: Distributionally Robust Learning of Shared Representations from Multi-Source Data

TL;DR

StablePCA introduces a distributionally robust approach to multi-source PCA, providing stable low-dimensional representations that are resistant to biases and systematic variations across sources, with theoretical guarantees and practical algorithms.

Contribution

We propose StablePCA, a novel distributionally robust PCA framework for multi-source data, including a convex relaxation, an efficient solver, and a data-dependent certificate for solution quality.

Findings

Convex relaxation of StablePCA is globally convergent.

The data-dependent certificate assesses the relaxation's tightness.

Alternative robust formulations based on different loss functions are explored.

Abstract

When synthesizing multi-source high-dimensional data, a key objective is to extract low-dimensional representations that effectively approximate the original features across different sources. Such representations facilitate the discovery of transferable structures and help mitigate systematic biases such as batch effects. We introduce Stable Principal Component Analysis (StablePCA), a distributionally robust framework for constructing stable latent representations by maximizing the worst-case explained variance over multiple sources. A primary challenge in extending classical PCA to the multi-source setting lies in the nonconvex rank constraint, which renders the StablePCA formulation a nonconvex optimization problem. To overcome this challenge, we conduct a convex relaxation of StablePCA and develop an efficient Mirror-Prox algorithm to solve the relaxed problem, with global convergence guarantees. Since the relaxed problem generally differs from the original formulation, we further introduce a data-dependent certificate to assess how well the algorithm solves the original nonconvex problem and establish the condition under which the relaxation is tight. Finally, we explore alternative distributionally robust formulations of multi-source PCA based on different loss functions.