Nonconvex Penalized LAD Estimation in Partial Linear Models with DNNs: Asymptotic Analysis and Proximal Algorithms

TL;DR

This paper develops a theoretical framework for penalized LAD estimation in partial linear models using DNNs, addressing nonconvexity, nonsmoothness, and high-dimensional challenges, and proposes proximal algorithms for computation.

Contribution

It introduces a novel asymptotic analysis for nonconvex penalized LAD with DNNs and proposes efficient proximal algorithms tailored for the complex optimization landscape.

Findings

Established consistency, convergence rate, and asymptotic normality of the estimator.

Analyzed the oracle problem and its continuous relaxation.

Proposed proximal subgradient methods with computational advantages.

Abstract

This paper investigates the partial linear model by Least Absolute Deviation (LAD) regression. We parameterize the nonparametric term using Deep Neural Networks (DNNs) and formulate a penalized LAD problem for estimation. Specifically, our model exhibits the following challenges. First, the regularization term can be nonconvex and nonsmooth, necessitating the introduction of infinite dimensional variational analysis and nonsmooth analysis into the asymptotic normality discussion. Second, our network must expand (in width, sparsity level and depth) as more samples are observed, thereby introducing additional difficulties for theoretical analysis. Third, the oracle of the proposed estimator is itself defined through a ultra high-dimensional, nonconvex, and discontinuous optimization problem, which already entails substantial computational and theoretical challenges. Under such the challenges, we establish the consistency, convergence rate, and asymptotic normality of the estimator. Furthermore, we analyze the oracle problem itself and its continuous relaxation. We study the convergence of a proximal subgradient method for both formulations, highlighting their structural differences lead to distinct computational subproblems along the iterations. In particular, the relaxed formulation admits significantly cheaper proximal updates, reflecting an inherent trade-off between statistical accuracy and computational tractability.