A Random Matrix Theory of Masked Self-Supervised Regression

TL;DR

This paper develops a high-dimensional theoretical framework using random matrix theory to analyze masked self-supervised learning, revealing phase transitions and conditions where SSL outperforms PCA.

Contribution

It introduces a novel high-dimensional analysis of masked SSL, deriving explicit formulas for generalization error and spectral structure, and identifies phase transitions in signal recovery.

Findings

Spectral analysis of the learned predictor reveals data structure extraction.

Identifies phase transition points where SSL recovers latent signals.

Shows regimes where SSL outperforms PCA in unsupervised learning.

Abstract

In the era of transformer models, masked self-supervised learning (SSL) has become a foundational training paradigm. A defining feature of masked SSL is that training aggregates predictions across many masking patterns, giving rise to a joint, matrix-valued predictor rather than a single vector-valued estimator. This object encodes how coordinates condition on one another and poses new analytical challenges. We develop a precise high-dimensional analysis of masked modeling objectives in the proportional regime where the number of samples scales with the ambient dimension. Our results provide explicit expressions for the generalization error and characterize the spectral structure of the learned predictor, revealing how masked modeling extracts structure from data. For spiked covariance models, we show that the joint predictor undergoes a Baik--Ben Arous--P\'ech\'e (BBP)-type phase transition, identifying when masked SSL begins to recover latent signals. Finally, we identify structured regimes in which masked self-supervised learning provably outperforms PCA, highlighting potential advantages of SSL objectives over classical unsupervised methods