Convex Programming for Estimation in Nonlinear Recurrent Models

TL;DR

This paper introduces a convex programming approach for estimating nonlinear recurrent models, including neural networks, providing theoretical guarantees and empirical validation through simulations.

Contribution

It presents a convex estimator for nonlinear recurrent models with proven sample complexity under stability assumptions, bridging theory and practical estimation.

Findings

Estimator performs well in simulations

Sample complexity bounds established for stable dynamics

Numerical experiments suggest robustness beyond theoretical assumptions

Abstract

We propose a formulation for nonlinear recurrent models that includes simple parametric models of recurrent neural networks as a special case. The proposed formulation leads to a natural estimator in the form of a convex program. We provide a sample complexity for this estimator in the case of stable dynamics, where the nonlinear recursion has a certain contraction property, and under certain regularity conditions on the input distribution. We evaluate the performance of the estimator by simulation on synthetic data. These numerical experiments also suggest the extent at which the imposed theoretical assumptions may be relaxed.