On Supervised On-line Rolling-Horizon Control for Infinite-Horizon Discounted Markov Decision Processes

TL;DR

This paper introduces an online, policy-iteration-based rolling-horizon control algorithm for infinite-horizon discounted MDPs, emphasizing convergence properties and incorporating supervisor feedback.

Contribution

It develops a novel asynchronous online algorithm that updates policies at visited states, differing from classical value iteration, and analyzes its convergence behavior.

Findings

Achieves global convergence to optimal policies under certain conditions.

Ensures local convergence to a fixed policy in finite time.

Incorporates supervisor feedback into policy updates.

Abstract

This note re-visits the rolling-horizon control approach to the problem of a Markov decision process (MDP) with infinite-horizon discounted expected reward criterion. Distinguished from the classical value-iteration approach, we develop an asynchronous on-line algorithm based on policy iteration integrated with a multi-policy improvement method of policy switching. A sequence of monotonically improving solutions to the forecast-horizon sub-MDP is generated by updating the current solution only at the currently visited state, building in effect a rolling-horizon control policy for the MDP over infinite horizon. Feedbacks from "supervisors," if available, can be also incorporated while updating. We focus on the convergence issue with a relation to the transition structure of the MDP. Either a global convergence to an optimal forecast-horizon policy or a local convergence to a "locally-optimal" fixed-policy in a finite time is achieved by the algorithm depending on the structure.