Beyond Vintage Rotation: Bias-Free Sparse Representation Learning with Oracle Inference

TL;DR

This paper introduces a bias-free rotation method for sparse representation learning that achieves oracle inference properties, enabling valid statistical inference in latent variable models.

Contribution

It proposes a novel bias-free rotation technique with theoretical guarantees, addressing bias issues in traditional vintage rotations for valid inference.

Findings

The new method achieves the same asymptotic variance as oracle estimators.

It provides statistically valid confidence intervals and hypothesis tests.

The computational framework retains the oracle property of estimators.

Abstract

Learning low-dimensional latent representations is a central topic in statistics and machine learning, and rotation methods have long been used to obtain sparse and interpretable representations. Despite nearly a century of widespread use across many fields, rigorous guarantees for valid inference for the learned representation remain lacking. In this paper, we identify a surprisingly prevalent phenomenon that suggests a reason for this gap: for a broad class of vintage rotations, the resulting estimators exhibit a non-estimable bias. Because this bias is independent of the data, it fundamentally precludes the development of valid inferential procedures, including the construction of confidence intervals and hypothesis testing. To address this challenge, we propose a novel bias-free rotation method within a general representation learning framework based on latent variables. We establish an oracle inference property for the learned sparse representations: the estimators achieve the same asymptotic variance as in the ideal setting where the latent variables are observed. To bridge the gap between theory and computation, we develop an efficient computational framework and prove that its output estimators retain the same oracle property. Our results provide a rigorous inference procedure for the rotated estimators, yielding statistically valid and interpretable representation learning.