A new perspective on the learning dynamics for a class of learning problems via averaged gradient systems coupled with diffusion-transmutation processes

TL;DR

This paper introduces a novel approach to understanding learning dynamics through averaged gradient systems coupled with diffusion-transmutation processes, providing insights into rare events and resampling-based learning methods.

Contribution

It develops a large deviation framework for coupled stochastic systems and offers a new perspective on learning dynamics using averaged gradient systems with resampled datasets.

Findings

Large deviation results for coupled stochastic processes.

A computational algorithm for variational problems related to rare events.

Numerical demonstration on nonlinear regression with random walk interpretation.

Abstract

In the first part of this paper, we consider a family of continuous-time dynamical systems coupled with diffusion-transmutation processes. Under certain conditions, such randomly perturbed dynamical systems can be interpreted as an averaged dynamical system, whose weighting coefficients, that depend on the state trajectory of the underlying averaged system, are assumed to be strictly positive with sum unity. Here, we provide a large deviation result for the corresponding family of processes, i.e., a variational problem formulation modeling the most likely sample path leading to certain noise-induced rare-events. This remarkably allows us to provide a computational algorithm for solving the corresponding variational problem. In the second part of the paper, we use some of the insights from the first part and provide a new perspective on the learning dynamics for a class of learning problems, whose averaged gradient dynamical systems, from continuous-time perspective, are guided by a set of subsampled datasets that are obtained from the original dataset via bootstrapping or other related resampling-based techniques. Finally, we present some numerical results for a typical nonlinear regression problem, where the corresponding averaged gradient system is interpreted as random walks on a graph, whose outgoing edges are uniformly chosen at random.