Selective Memory Recursive Least Squares: Recast Forgetting into Memory in RBF Neural Network Based Real-Time Learning

TL;DR

This paper introduces SMRLS, a novel real-time training method for RBF neural networks that replaces forgetting mechanisms with a memory-based approach, enhancing learning speed and generalization by considering both temporal and spatial data importance.

Contribution

The paper proposes a new SMRLS method that recasts forgetting into a memory mechanism, improving real-time learning in RBF neural networks by considering data distribution.

Findings

SMRLS outperforms classical methods like FFRLS and SGD in learning speed.

SMRLS improves generalization capability in RBFNNs.

Simulation results validate the effectiveness of SMRLS.

Abstract

In radial basis function neural network (RBFNN) based real-time learning tasks, forgetting mechanisms are widely used such that the neural network can keep its sensitivity to new data. However, with forgetting mechanisms, some useful knowledge will get lost simply because they are learned a long time ago, which we refer to as the passive knowledge forgetting phenomenon. To address this problem, this paper proposes a real-time training method named selective memory recursive least squares (SMRLS) in which the classical forgetting mechanisms are recast into a memory mechanism. Different from the forgetting mechanism, which mainly evaluates the importance of samples according to the time when samples are collected, the memory mechanism evaluates the importance of samples through both temporal and spatial distribution of samples. With SMRLS, the input space of the RBFNN is evenly divided into a finite number of partitions and a synthesized objective function is developed using synthesized samples from each partition. In addition to the current approximation error, the neural network also updates its weights according to the recorded data from the partition being visited. Compared with classical training methods including the forgetting factor recursive least squares (FFRLS) and stochastic gradient descent (SGD) methods, SMRLS achieves improved learning speed and generalization capability, which are demonstrated by corresponding simulation results.