Strategies in POMDPs with Stage Duration

TL;DR

This paper studies POMDPs with stage duration, showing that strategies can be transferred between different stage durations, and proves the existence of a continuous-time limit of the asymptotic value.

Contribution

It introduces a method to mimic strategies across different stage durations and establishes the monotonicity and limit of the value function as stage duration approaches zero.

Findings

Strategies in POMDPs with stage duration can be replicated in the base POMDP.

The value function V(h) is nondecreasing with respect to h.

The limit of V(h) as h approaches zero exists.

Abstract

Partially observable Markov decision processes (POMDPs) with stage duration provide a framework for approximating continuous-time behavior by scaling transition probabilities with a stage duration parameter $h \in (0,1]$. While previous literature has primarily focused on the limit of the discounted value as the stage duration $h$ vanishes, this paper investigates the global behavior of the asymptotic value, $V(h)$, across varying stage durations. Our main result demonstrates that any strategy in a POMDP with stage duration $h$ can be mimicked in the base POMDP ($h=1$). Specifically, we provide an explicit construction showing that for any strategy in the POMDP with stage duration $h$, there exists a strategy in the base POMDP that secures the same asymptotic payoff. As a consequence of this theorem, we establish that the value function $V(h)$ is nondecreasing with respect to $h$, and that the continuous-time limit $\lim_{h \to 0} V(h)$ exists.