Missingness-Adaptive Factor Identification in High-Dimensional Data

TL;DR

This paper presents MATE, a novel missingness-adaptive estimator for determining the number of factors in high-dimensional, incomplete data, without imputation, and with proven consistency.

Contribution

Introduces MATE, the first framework that adaptively handles missing data for factor number estimation without restrictive assumptions or imputation.

Findings

MATE outperforms existing methods in high missingness scenarios.

MATE is consistent under various structural conditions.

Extensive simulations and real data validate MATE's robustness.

Abstract

Determining the number of factors in high-dimensional factor models remains a fundamental challenge, particularly when data are incomplete. This paper introduces the concept of identifiable factors, those that can be reliably recovered despite missing observations, and proposes the Missingness-Adaptive Thresholding Estimator (MATE). To our knowledge, MATE is the first missingness-adaptive framework for factor number determination that accommodates both homogeneous and heterogeneous missingness without imposing restrictive assumptions on factor strength. Notably, it operates without data imputation, circumventing the computational burden associated with most existing approaches. We establish a rigorous theoretical foundation for MATE, proving its consistency under a range of structural conditions. Extensive simulations and real-world applications demonstrate that MATE consistently outperforms state-of-the-art methods, exhibiting superior robustness in settings with high missingness rates and weak factor signals.