Embedding generalization within the learning dynamics: An approach based-on sample path large deviation theory

TL;DR

This paper introduces a novel approach to enhance generalization in learning models by analyzing the learning dynamics through sample path large deviation theory, linking stochastic gradient processes to optimal control and rare event probabilities.

Contribution

It develops a theoretical framework using Freidlin-Wentzell large deviations to estimate the probability of hitting target loss landscapes, connecting learning dynamics with optimal control and robustness.

Findings

Provides asymptotic probability estimates for rare events in learning dynamics.

Establishes a connection between large deviations and optimal control in model training.

Offers a computational algorithm for optimal point estimation in nonlinear regression.

Abstract

We consider a typical learning problem of point estimations for modeling of nonlinear functions or dynamical systems in which generalization, i.e., verifying a given learned model, can be embedded as an integral part of the learning process or dynamics. In particular, we consider an empirical risk minimization based learning problem that exploits gradient methods from continuous-time perspective with small random perturbations, which is guided by the training dataset loss. Here, we provide an asymptotic probability estimate in the small noise limit based-on the Freidlin-Wentzell theory of large deviations, when the sample path of the random process corresponding to the randomly perturbed gradient dynamical system hits a certain target set, i.e., a rare event, when the latter is specified by the testing dataset loss landscape. Interestingly, the proposed framework can be viewed as one way of improving generalization and robustness in learning problems that provides new insights leading to optimal point estimates which is guided by training data loss, while, at the same time, the learning dynamics has an access to the testing dataset loss landscape in some form of future achievable or anticipated target goal. Moreover, as a by-product, we establish a connection with optimal control problem, where the target set, i.e., the rare event, is considered as the desired outcome or achievable target goal for a certain optimal control problem, for which we also provide a verification result reinforcing the rationale behind the proposed framework. Finally, we present a computational algorithm that solves the corresponding variational problem leading to an optimal point estimates and, as part of this work, we also present some numerical results for a typical case of nonlinear regression problem.