Minimax Regret for Stochastic Shortest Path

TL;DR

This paper establishes near-optimal regret bounds for learning in stochastic shortest path problems, introducing a reduction to finite-horizon MDPs and providing new theoretical insights into the problem's complexity.

Contribution

The authors derive tight minimax regret bounds for SSP with unknown costs, and propose a novel reduction to finite-horizon MDPs that facilitates analysis.

Findings

Matching upper and lower regret bounds for B* ≥ 1

Improved regret bounds by a factor of √|S| over previous work

Algorithm for finite-horizon MDPs with polynomial dependence on expected cost

Abstract

We study the Stochastic Shortest Path (SSP) problem in which an agent has to reach a goal state in minimum total expected cost. In the learning formulation of the problem, the agent has no prior knowledge about the costs and dynamics of the model. She repeatedly interacts with the model for $K$ episodes, and has to minimize her regret. In this work we show that the minimax regret for this setting is $\widetilde O(\sqrt{ (B_\star^2 + B_\star) |S| |A| K})$ where $B_\star$ is a bound on the expected cost of the optimal policy from any state, $S$ is the state space, and $A$ is the action space. This matches the $\Omega (\sqrt{ B_\star^2 |S| |A| K})$ lower bound of Rosenberg et al. [2020] for $B_\star \ge 1$, and improves their regret bound by a factor of $\sqrt{|S|}$. For $B_\star < 1$ we prove a matching lower bound of $\Omega (\sqrt{ B_\star |S| |A| K})$. Our algorithm is based on a novel reduction from SSP to finite-horizon MDPs. To that end, we provide an algorithm for the finite-horizon setting whose leading term in the regret depends polynomially on the expected cost of the optimal policy and only logarithmically on the horizon.