Some Simulation Results for Emphatic Temporal-Difference Learning Algorithms

TL;DR

This paper provides simulation results demonstrating the behavior of constrained emphatic temporal-difference (ETD) algorithms across three example problems, complementing previous theoretical convergence analyses.

Contribution

It offers empirical insights into ETD algorithms' performance, supplementing prior theoretical work with practical simulation data.

Findings

ETD algorithms exhibit stable behavior in simulated environments

Simulation results align with theoretical convergence properties

Behavior varies across different example problems

Abstract

This is a companion note to our recent study of the weak convergence properties of constrained emphatic temporal-difference learning (ETD) algorithms from a theoretic perspective. It supplements the latter analysis with simulation results and illustrates the behavior of some of the ETD algorithms using three example problems.