Off-Policy Temporal Difference Learning for Perturbed Markov Decision Processes: Theoretical Insights and Extensive Simulations

TL;DR

This paper introduces an off-policy Temporal Difference learning method for perturbed Markov Decision Processes, providing theoretical insights and extensive simulations to address environmental uncertainties and high-dimensional challenges.

Contribution

It develops a novel off-policy TD approach that maintains contraction properties under perturbations and demonstrates its effectiveness through theoretical analysis and simulations.

Findings

The method preserves contraction mapping in perturbed environments.

The approach effectively handles large state and action spaces.

Simulations validate the theoretical advantages of the proposed method.

Abstract

Dynamic Programming suffers from the curse of dimensionality due to large state and action spaces, a challenge further compounded by uncertainties in the environment. To mitigate these issue, we explore an off-policy based Temporal Difference Approximate Dynamic Programming approach that preserves contraction mapping when projecting the problem into a subspace of selected features, accounting for the probability distribution of the perturbed transition probability matrix. We further demonstrate how this Approximate Dynamic Programming approach can be implemented as a particular variant of the Temporal Difference learning algorithm, adapted for handling perturbations. To validate our theoretical findings, we provide a numerical example using a Markov Decision Process corresponding to a resource allocation problem.