A Constrained Randomized Shortest-Paths Framework for Optimal Exploration

TL;DR

This paper extends the randomized shortest-paths framework to include equality constraints, providing algorithms for optimal exploration and exploitation balancing in networks and Markov decision processes.

Contribution

It introduces a generic constrained RSP algorithm using Lagrangian duality and a simple iterative method for computing optimal randomized policies.

Findings

Algorithms effectively balance exploration and exploitation.

The approach generalizes soft Bellman-Ford and value iteration.

Simulation confirms the model's expected behavior.

Abstract

The present work extends the randomized shortest-paths framework (RSP), interpolating between shortest-path and random-walk routing in a network, in three directions. First, it shows how to deal with equality constraints on a subset of transition probabilities and develops a generic algorithm for solving this constrained RSP problem using Lagrangian duality. Second, it derives a surprisingly simple iterative procedure to compute the optimal, randomized, routing policy generalizing the previously developed "soft" Bellman-Ford algorithm. The resulting algorithm allows balancing exploitation and exploration in an optimal way by interpolating between a pure random behavior and the deterministic, optimal, policy (least-cost paths) while satisfying the constraints. Finally, the two algorithms are applied to Markov decision problems by considering the process as a constrained RSP on a bipartite state-action graph. In this context, the derived "soft" value iteration algorithm appears to be closely related to dynamic policy programming as well as Kullback-Leibler and path integral control, and similar to a recently introduced reinforcement learning exploration strategy. This shows that this strategy is optimal in the RSP sense - it minimizes expected path cost subject to relative entropy constraint. Simulation results on illustrative examples show that the model behaves as expected.