Regret Guarantees for Linear Contextual Stochastic Shortest Path

TL;DR

This paper introduces LR-CSSP, an algorithm for linear contextual stochastic shortest path problems, providing regret guarantees and handling continuous contexts with all episodes terminating efficiently.

Contribution

The paper proposes LR-CSSP, a novel algorithm with regret bounds for linear CSSP, addressing challenges of unknown dynamics and continuous contexts in stochastic shortest path problems.

Findings

LR-CSSP achieves sublinear regret bounds.

The algorithm handles continuous context spaces effectively.

All episodes terminate within a reasonable timeframe.

Abstract

We define the problem of linear Contextual Stochastic Shortest Path (CSSP), where at the beginning of each episode, the learner observes an adversarially chosen context that determines the MDP through a fixed but unknown linear function. The learner's objective is to reach a designated goal state with minimal expected cumulative loss, despite having no prior knowledge of the transition dynamics, loss functions, or the mapping from context to MDP. In this work, we propose LR-CSSP, an algorithm that achieves a regret bound of $\widetilde{O}(K^{2/3} d^{2/3} |S| |A|^{1/3} B_\star^2 T_\star \log (1/ \delta))$, where $K$ is the number of episodes, $d$ is the context dimension, $S$ and $A$ are the sets of states and actions respectively, $B_\star$ bounds the optimal cumulative loss and $T_\star$, unknown to the learner, bounds the expected time for the optimal policy to reach the goal. In the case where all costs exceed $\ell_{\min}$, LR-CSSP attains a regret of $\widetilde O(\sqrt{K \cdot d^2 |S|^3 |A| B_\star^3 \log(1/\delta)/\ell_{\min}})$. Unlike in contextual finite-horizon MDPs, where limited knowledge primarily leads to higher losses and regret, in the CSSP setting, insufficient knowledge can also prolong episodes and may even lead to non-terminating episodes. Our analysis reveals that LR-CSSP effectively handles continuous context spaces, while ensuring all episodes terminate within a reasonable number of time steps.