Stochastic Shortest Path with Adversarially Changing Costs

TL;DR

This paper introduces the adversarial stochastic shortest path (SSP) model where costs change adversarially over time, and develops algorithms with provable regret bounds, advancing the understanding of SSP in adversarial environments.

Contribution

First algorithms for adversarial SSP with regret bounds, addressing a natural setting with changing costs and unknown transitions.

Findings

Achieved high probability regret bounds of ~O(√K) for positive costs.

Established regret bounds of ~O(K^{3/4}) in the general case.

First to analyze adversarial SSP with sub-linear regret guarantees.

Abstract

Stochastic shortest path (SSP) is a well-known problem in planning and control, in which an agent has to reach a goal state in minimum total expected cost. In this paper we present the adversarial SSP model that also accounts for adversarial changes in the costs over time, while the underlying transition function remains unchanged. Formally, an agent interacts with an SSP environment for $K$ episodes, the cost function changes arbitrarily between episodes, and the transitions are unknown to the agent. We develop the first algorithms for adversarial SSPs and prove high probability regret bounds of $\widetilde O (\sqrt{K})$ assuming all costs are strictly positive, and $\widetilde O (K^{3/4})$ in the general case. We are the first to consider this natural setting of adversarial SSP and obtain sub-linear regret for it.