Recurrent Neural Network Training with Convex Loss and Regularization Functions by Extended Kalman Filtering

TL;DR

This paper introduces a novel training method for recurrent neural networks using extended Kalman filtering, capable of handling convex loss functions and regularization, showing competitive performance with stochastic gradient descent.

Contribution

The paper presents a new Kalman filtering-based approach for training RNNs with convex loss and regularization, including $ ext{L}_1$-regularization, and demonstrates its effectiveness in system identification and control tasks.

Findings

Competitive with stochastic gradient descent in benchmarks

Effective in training with $ ext{L}_1$-regularization

Applicable to nonlinear model predictive control

Abstract

This paper investigates the use of extended Kalman filtering to train recurrent neural networks with rather general convex loss functions and regularization terms on the network parameters, including $\ell_1$-regularization. We show that the learning method is competitive with respect to stochastic gradient descent in a nonlinear system identification benchmark and in training a linear system with binary outputs. We also explore the use of the algorithm in data-driven nonlinear model predictive control and its relation with disturbance models for offset-free closed-loop tracking.