$L_0$ regularized estimation for nonlinear models that have sparse underlying linear structures

TL;DR

This paper develops $L_0$ regularized estimation methods for nonlinear models with sparse underlying linear structures, providing error bounds under certain conditions, applicable to exponential models and analytic functions.

Contribution

It introduces a unified approach to bound the $L_2$ error of $L_0$ regularized estimators in nonlinear models with sparse parameters, covering exponential and analytic functions.

Findings

Derived $L_2$ error bounds for $L_0$ regularized MLE in exponential models.

Established $L_2$ error bounds for $L_0$ regularized least squares with analytic functions.

Provided conditions under which the $L_0$ regularized estimators perform reliably.

Abstract

We study the estimation of $\beta$ for the nonlinear model $y = f(X\sp{\top}\beta) + \epsilon$ when $f$ is a nonlinear transformation that is known, $\beta$ has sparse nonzero coordinates, and the number of observations can be much smaller than that of parameters ($n\ll p$). We show that in order to bound the $L_2$ error of the $L_0$ regularized estimator $\hat\beta$, i.e., $\|\hat\beta - \beta\|_2$, it is sufficient to establish two conditions. Based on this, we obtain bounds of the $L_2$ error for (1) $L_0$ regularized maximum likelihood estimation (MLE) for exponential linear models and (2) $L_0$ regularized least square (LS) regression for the more general case where $f$ is analytic. For the analytic case, we rely on power series expansion of $f$, which requires taking into account the singularities of $f$.