Convex duality for stochastic shortest path problems in known and unknown environments

TL;DR

This paper explores convex duality in stochastic shortest path problems, extending value iteration operators to unknown environments and analyzing their duals and convergence properties.

Contribution

It introduces new convex optimization formulations for SSPs with unknown parameters and examines their duals, revealing complex convergence behaviors and open research questions.

Findings

Extended value iteration operators relate to convex programs and their duals.

Finite horizon bounds can be applied but may lead to non-monotone operators.

Oscillating behavior observed in certain special cases.

Abstract

This paper studies Stochastic Shortest Path (SSP) problems in known and unknown environments from the perspective of convex optimisation. It first recalls results in the known parameter case, and develops understanding through different proofs. It then focuses on the unknown parameter case, where it studies extended value iteration (EVI) operators. This includes the existing operators used in Rosenberg et al. [26] and Tarbouriech et al. [31] based on the l-1 norm and supremum norm, as well as defining EVI operators corresponding to other norms and divergences, such as the KL-divergence. This paper shows in general how the EVI operators relate to convex programs, and the form of their dual, where strong duality is exhibited. This paper then focuses on whether the bounds from finite horizon research of Neu and Pike-Burke [21] can be applied to these extended value iteration operators in the SSP setting. It shows that similar bounds to [21] for these operators exist, however they lead to operators that are not in general monotone and have more complex convergence properties. In a special case we observe oscillating behaviour. This paper generates open questions on how research may progress, with several examples that require further examination.