Online Reinforcement Learning of Optimal Threshold Policies for Markov Decision Processes

TL;DR

This paper introduces a structure-aware reinforcement learning algorithm that exploits the multi-threshold structure of optimal policies in Markov Decision Processes, leading to faster convergence and reduced computational complexity.

Contribution

The paper proposes a novel RL algorithm that leverages the threshold structure of optimal policies in MDPs, with proven convergence and improved efficiency over traditional methods.

Findings

Faster convergence compared to classical RL algorithms.

Reduced storage and computational complexity.

Effective exploitation of policy structure enhances learning efficiency.

Abstract

To overcome the curses of dimensionality and modeling of Dynamic Programming (DP) methods to solve Markov Decision Process (MDP) problems, Reinforcement Learning (RL) methods are adopted in practice. Contrary to traditional RL algorithms which do not consider the structural properties of the optimal policy, we propose a structure-aware learning algorithm to exploit the ordered multi-threshold structure of the optimal policy, if any. We prove the asymptotic convergence of the proposed algorithm to the optimal policy. Due to the reduction in the policy space, the proposed algorithm provides remarkable improvements in storage and computational complexities over classical RL algorithms. Simulation results establish that the proposed algorithm converges faster than other RL algorithms.