Structural Risk Minimization for Learning Nonlinear Dynamics

TL;DR

This paper introduces a novel Structural Risk Minimization framework for learning nonlinear dynamics, balancing model complexity and learnability, with theoretical guarantees and empirical validation across kernel and neural network classes.

Contribution

It proposes a new SRM approach for nonlinear dynamics, providing explicit schemes and bounds for hierarchies of models, improving model selection and learning guarantees.

Findings

SRM bounds track relative true prediction error across model classes

Explicit SRM schemes derived for kernel and neural network hierarchies

Empirical results support the effectiveness of the SRM bounds in model selection

Abstract

Recent advances in learning or identification of nonlinear dynamics focus on learning a suitable model within a pre-specified model class. However, a key difficulty that remains is the choice of the model class from which the dynamics will be learned. The fundamental challenge is trading the richness of the model class with the learnability within the model class. Toward addressing the so-called model selection problem, we introduce a novel notion of Structural Risk Minimization (SRM) for learning nonlinear dynamics. Inspired by classical SRM for classification, we minimize a bound on the true prediction error over hierarchies of model classes. The class selected by our SRM scheme is shown to achieve a nearly optimal learning guarantee among all model classes contained in the hierarchy. Employing the proposed scheme along with computable model class complexity bounds, we derive explicit SRM schemes for learning nonlinear dynamics under hierarchies of: i) norm-constrained Reproducing Kernel Hilbert Spaces, and ii) norm-constrained Neural Network classes. We empirically show that even though too loose to be used as absolute estimates, our SRM bounds on the true prediction error are able to track its relative behavior across different model classes of the hierarchy.