RNN-based Online Learning: An Efficient First-Order Optimization Algorithm with a Convergence Guarantee

TL;DR

This paper presents a new online learning algorithm for RNNs that guarantees convergence without assumptions on the environment, demonstrating improved performance over existing methods through simulations.

Contribution

It introduces an efficient first-order online training algorithm for RNNs with proven convergence guarantees, applicable in any learning environment.

Findings

Algorithm converges to optimal parameters in online RNN training.

Numerical simulations confirm theoretical convergence and performance improvements.

Outperforms state-of-the-art RNN training methods in experiments.

Abstract

We investigate online nonlinear regression with continually running recurrent neural network networks (RNNs), i.e., RNN-based online learning. For RNN-based online learning, we introduce an efficient first-order training algorithm that theoretically guarantees to converge to the optimum network parameters. Our algorithm is truly online such that it does not make any assumption on the learning environment to guarantee convergence. Through numerical simulations, we verify our theoretical results and illustrate significant performance improvements achieved by our algorithm with respect to the state-of-the-art RNN training methods.