Pontryagin's Minimum Principle and Forward-Backward Sweep Method for the System of HJB-FP Equations in Memory-Limited Partially Observable Stochastic Control

TL;DR

This paper introduces a novel interpretation of the coupled HJB-FP equations in memory-limited partially observable stochastic control via Pontryagin's minimum principle, and proposes a forward-backward sweep method with guaranteed convergence for this setting.

Contribution

It provides a new theoretical interpretation of HJB-FP equations and develops a convergent forward-backward sweep algorithm for ML-POSC.

Findings

FBSM converges in ML-POSC due to limited coupling.

The interpretation links HJB-FP system to Pontryagin's principle.

Algorithm effectively computes optimal control in memory-limited scenarios.

Abstract

Memory-limited partially observable stochastic control (ML-POSC) is the stochastic optimal control problem under incomplete information and memory limitation. In order to obtain the optimal control function of ML-POSC, a system of the forward Fokker-Planck (FP) equation and the backward Hamilton-Jacobi-Bellman (HJB) equation needs to be solved. In this work, we firstly show that the system of HJB-FP equations can be interpreted via the Pontryagin's minimum principle on the probability density function space. Based on this interpretation, we then propose the forward-backward sweep method (FBSM) to ML-POSC, which has been used in the Pontryagin's minimum principle. FBSM is an algorithm to compute the forward FP equation and the backward HJB equation alternately. Although the convergence of FBSM is generally not guaranteed, it is guaranteed in ML-POSC because the coupling of HJB-FP equations is limited to the optimal control function in ML-POSC.