Convergent Reinforcement Learning Algorithms for Stochastic Shortest Path Problem

TL;DR

This paper introduces new convergent reinforcement learning algorithms for the stochastic shortest path problem, demonstrating their effectiveness in both tabular and function approximation settings with superior and reliable performance.

Contribution

It presents two novel algorithms for SSP in tabular settings and one for function approximation, all with proven convergence and improved performance.

Findings

Tabular algorithms outperform existing RL algorithms in SSP.

Function approximation algorithm shows reliable performance.

All algorithms demonstrate asymptotic almost-sure convergence.

Abstract

In this paper we propose two algorithms in the tabular setting and an algorithm for the function approximation setting for the Stochastic Shortest Path (SSP) problem. SSP problems form an important class of problems in Reinforcement Learning (RL), as other types of cost-criteria in RL can be formulated in the setting of SSP. We show asymptotic almost-sure convergence for all our algorithms. We observe superior performance of our tabular algorithms compared to other well-known convergent RL algorithms. We further observe reliable performance of our function approximation algorithm compared to other algorithms in the function approximation setting.