Sparse Nonlinear Regression: Parameter Estimation and Asymptotic Inference

TL;DR

This paper introduces a method for estimating parameters and conducting inference in sparse nonlinear regression models, proving convergence properties and providing statistical guarantees despite the nonconvex optimization landscape.

Contribution

It proposes an $ ext{l}_1$-regularized estimator for nonlinear regression, demonstrating its statistical optimality and developing an efficient algorithm with inference capabilities.

Findings

Stationary points achieve optimal convergence rates

Algorithm converges to a stationary point efficiently

Valid hypothesis tests and confidence intervals are constructed

Abstract

We study parameter estimation and asymptotic inference for sparse nonlinear regression. More specifically, we assume the data are given by $y = f( x^\top \beta^* ) + \epsilon$, where $f$ is nonlinear. To recover $\beta^*$, we propose an $\ell_1$-regularized least-squares estimator. Unlike classical linear regression, the corresponding optimization problem is nonconvex because of the nonlinearity of $f$. In spite of the nonconvexity, we prove that under mild conditions, every stationary point of the objective enjoys an optimal statistical rate of convergence. In addition, we provide an efficient algorithm that provably converges to a stationary point. We also access the uncertainty of the obtained estimator. Specifically, based on any stationary point of the objective, we construct valid hypothesis tests and confidence intervals for the low dimensional components of the high-dimensional parameter $\beta^*$. Detailed numerical results are provided to back up our theory.