Reinforcement Learning for Optimal Stopping in POMDPs with Application to Quickest Change Detection

TL;DR

This paper applies reinforcement learning, specifically Q-learning, to the optimal stopping problem in POMDPs for quickest change detection, providing a new approach with convergence guarantees and practical effectiveness.

Contribution

It introduces a Q-learning algorithm for partially observed optimal stopping problems, with convergence analysis and application to quickest change detection.

Findings

Q-learning converges under linear function approximation

The proposed policies are near-optimal in several scenarios

Numerical experiments demonstrate effective performance close to theoretical best

Abstract

The field of quickest change detection (QCD) focuses on the design and analysis of online algorithms that estimate the time at which a significant event occurs. In this paper, design and analysis are cast in a Bayesian framework, where QCD is formulated as an optimal stopping problem with partial observations. An approximately optimal detection algorithm is sought using techniques from reinforcement learning. The contributions of the paper are summarized as follows: (i) A Q-learning algorithm is proposed for the general partially observed optimal stopping problem. It is shown to converge under linear function approximation, given suitable assumptions on the basis functions. An example is provided to demonstrate that these assumptions are necessary to ensure algorithmic stability. (ii) Prior theory motivates a particular choice of features in applying Q-learning to QCD. It is shown that, in several scenarios and under ideal conditions, the resulting class of policies contains one that is approximately optimal. (iii) Numerical experiments show that Q-learning consistently produces policies that perform close to the best achievable within the chosen function class.