Fast and Reliable Parameter Estimation from Nonlinear Observations

TL;DR

This paper introduces a framework to analyze the tradeoffs between data, computation, and prior knowledge in estimating parameters from nonlinear observations, providing convergence guarantees for common algorithms.

Contribution

It offers a unified analysis of various algorithms for nonlinear parameter estimation, including convergence rates and sample complexity, accounting for unknown nonlinear functions.

Findings

Projected gradient descent converges linearly with minimal samples

Characterization of convergence rate based on nonlinearity measure

Tradeoff analysis between data, computation, and prior knowledge

Abstract

In this paper we study the problem of recovering a structured but unknown parameter ${\bf{\theta}}^*$ from $n$ nonlinear observations of the form $y_i=f(\langle {\bf{x}}_i,{\bf{\theta}}^*\rangle)$ for $i=1,2,\ldots,n$. We develop a framework for characterizing time-data tradeoffs for a variety of parameter estimation algorithms when the nonlinear function $f$ is unknown. This framework includes many popular heuristics such as projected/proximal gradient descent and stochastic schemes. For example, we show that a projected gradient descent scheme converges at a linear rate to a reliable solution with a near minimal number of samples. We provide a sharp characterization of the convergence rate of such algorithms as a function of sample size, amount of a-prior knowledge available about the parameter and a measure of the nonlinearity of the function $f$. These results provide a precise understanding of the various tradeoffs involved between statistical and computational resources as well as a-prior side information available for such nonlinear parameter estimation problems.