A Pontryagin Maximum Principle on the Belief Space for Continuous-Time Optimal Control with Discrete Observations

TL;DR

This paper develops a Pontryagin maximum principle on the belief space for continuous-time stochastic control with discrete observations, linking optimality conditions to nonlinear filtering equations and providing a particle-based numerical solution.

Contribution

It introduces a novel maximum principle on the belief space for hybrid continuous-discrete control problems, connecting it to filtering equations and proposing a practical particle filtering algorithm.

Findings

The maximum principle provides necessary conditions for optimal control under partial observations.

The relationship between the adjoint process and the value functional gradient is established.

Numerical experiments demonstrate the effectiveness of the particle-based control scheme.

Abstract

We study a continuous time stochastic optimal control problem under partial observations that are available only at discrete time instants. This hybrid setting, with continuous dynamics and intermittent noisy measurements, arises in applications ranging from robotic exploration and target tracking to epidemic control. We formulate the problem on the space of beliefs (information states), treating the controller's posterior distribution of the state as the state variable for decision making. On this belief space we derive a Pontryagin maximum principle that provides necessary conditions for optimality. The analysis carefully tracks both the continuous evolution of the state between observation times and the Bayesian jump updates of the belief at observation instants. A key insight is a relationship between the adjoint process in our maximum principle and the gradient of the value functional on the belief space, which links the optimality conditions to the dynamic programming approach on the space of probability measures. The resulting optimality system has a prediction and update structure that is closely related to the unnormalised Zakai equation and the normalised Kushner-Stratonovich equation in nonlinear filtering. Building on this analysis, we design a particle based numerical scheme to approximate the coupled forward (filter) and backward (adjoint) system. The scheme uses particle filtering to represent the evolving belief and regression techniques to approximate the adjoint, which yields a practical algorithm for computing near optimal controls under partial information. The effectiveness of the approach is illustrated on both linear and nonlinear examples and highlights in particular the benefits of actively controlling the observation process.