Sensitivity of Filter Kernels and Robustness Bounds to Transition and Measurement Kernel Perturbations in Partially Observable Stochastic Control

TL;DR

This paper derives explicit bounds on the performance loss in POMDPs due to perturbations in transition and observation kernels, providing theoretical guarantees for robustness and model reduction techniques.

Contribution

It offers the first explicit bounds on value differences in POMDPs under kernel perturbations, including quantized models, with performance guarantees.

Findings

Explicit bounds on value differences using Wasserstein and total variation distances.

Control policies remain effective under model perturbations with quantifiable performance loss.

Error bounds decrease as model approximations become finer.

Abstract

Studying the stability of partially observed Markov decision processes (POMDPs) with respect to perturbations in either transition or observation kernels is a significant problem. While asymptotic robustness/stability results as approximate transition kernels and/or measurement kernels converge to the true ones have been previously reported, studies on explicit bounds on value differences and mismatch costs have been limited in scope for POMDPs. In this paper, we provide such explicit bounds under both discounted and average cost criteria. To this end, and also as an independent contribution, we first study the perturbations induced on the filter kernels (that is, the kernels of the belief-MDP reduction of POMDPs) as the transition and measurement kernels are perturbed. The bounds are given in terms of Wasserstein and total variation distances between the original and approximate transition and observation kernels. We then show that control policies optimized for approximate models yield performance guarantees when applied to the true model with explicit bounds. As a particular application, we consider the case where the state space and the measurement spaces are quantized to obtain finite models, and we obtain explicit error bounds which decay to zero as the approximations get finer. This provides explicit performance guarantees for model reduction in POMDPs.