Kalman filter control in the reinforcement learning framework

TL;DR

This paper demonstrates how to adapt Kalman-filter models for reinforcement learning, enabling online optimal control estimation with a Hebbian learning rule for value updates, bridging control theory and learning algorithms.

Contribution

It introduces a modification to the linear-quadratic-Gaussian Kalman-filter model that allows online control estimation and integrates reinforcement learning principles.

Findings

Enables online estimation of optimal control using Kalman filters.

Introduces a Hebbian learning rule for value estimation.

Bridges Kalman filtering with reinforcement learning frameworks.

Abstract

There is a growing interest in using Kalman-filter models in brain modelling. In turn, it is of considerable importance to make Kalman-filters amenable for reinforcement learning. In the usual formulation of optimal control it is computed off-line by solving a backward recursion. In this technical note we show that slight modification of the linear-quadratic-Gaussian Kalman-filter model allows the on-line estimation of optimal control and makes the bridge to reinforcement learning. Moreover, the learning rule for value estimation assumes a Hebbian form weighted by the error of the value estimation.