Nonsparse learning with latent variables

TL;DR

This paper introduces a novel nonsparse learning methodology that accounts for latent variables, improving model interpretability and accuracy by simultaneously recovering observable predictors and unobservable factors.

Contribution

The paper proposes a new NSL method that incorporates latent variables, providing theoretical guarantees and practical algorithms for better model selection and prediction.

Findings

Convergence rates for principal components and score vectors are established.

Model selection consistency and oracle inequalities are proven.

Simulation and real data demonstrate the effectiveness of the method.

Abstract

As a popular tool for producing meaningful and interpretable models, large-scale sparse learning works efficiently when the underlying structures are indeed or close to sparse. However, naively applying the existing regularization methods can result in misleading outcomes due to model misspecification. In particular, the direct sparsity assumption on coefficient vectors has been questioned in real applications. Therefore, we consider nonsparse learning with the conditional sparsity structure that the coefficient vector becomes sparse after taking out the impacts of certain unobservable latent variables. A new methodology of nonsparse learning with latent variables (NSL) is proposed to simultaneously recover the significant observable predictors and latent factors as well as their effects. We explore a common latent family incorporating population principal components and derive the convergence rates of both sample principal components and their score vectors that hold for a wide class of distributions. With the properly estimated latent variables, properties including model selection consistency and oracle inequalities under various prediction and estimation losses are established for the proposed methodology. Our new methodology and results are evidenced by simulation and real data examples.