Finding the Stochastic Shortest Path with Low Regret: The Adversarial Cost and Unknown Transition Case

TL;DR

This paper advances the stochastic shortest path problem by developing algorithms with low regret for adversarial costs and unknown transitions, improving existing bounds and addressing the most challenging feedback and transition scenarios.

Contribution

It introduces new algorithms with improved regret bounds for adversarial shortest path problems under unknown transitions, including the first solution for bandit feedback with adversarial costs and unknown transitions.

Findings

Achieves $ ilde{O}( ext{sqrt}(S^2ADT_ ext{star}K))$ regret in full-information setting.

Achieves $ ilde{O}( ext{sqrt}(S^3A^2DT_ ext{star}K))$ regret in bandit feedback setting.

Provides near-optimal regret algorithms for the hardest case of bandit feedback with adversarial costs and unknown transitions.

Abstract

We make significant progress toward the stochastic shortest path problem with adversarial costs and unknown transition. Specifically, we develop algorithms that achieve $\widetilde{O}(\sqrt{S^2ADT_\star K})$ regret for the full-information setting and $\widetilde{O}(\sqrt{S^3A^2DT_\star K})$ regret for the bandit feedback setting, where $D$ is the diameter, $T_\star$ is the expected hitting time of the optimal policy, $S$ is the number of states, $A$ is the number of actions, and $K$ is the number of episodes. Our work strictly improves (Rosenberg and Mansour, 2020) in the full information setting, extends (Chen et al., 2020) from known transition to unknown transition, and is also the first to consider the most challenging combination: bandit feedback with adversarial costs and unknown transition. To remedy the gap between our upper bounds and the current best lower bounds constructed via a stochastically oblivious adversary, we also propose algorithms with near-optimal regret for this special case.