True Online TD-Replan(lambda) Achieving Planning through Replaying

TL;DR

This paper introduces True Online TD-Replan(λ), a novel planning method that enhances experience replay efficiency by integrating replay density control via the λ parameter, outperforming existing methods in benchmark tests.

Contribution

The paper presents a new planning algorithm that extends true online TD(λ) with experience replay capabilities controlled by λ, improving performance over similar quadratic complexity methods.

Findings

Outperforms true online TD(λ) in experience replay tasks

Surpasses Dyna Planning and TD(λ)-Replan algorithms in benchmarks

Effective in both simple and complex feature environments

Abstract

In this paper, we develop a new planning method that extends the capabilities of the true online TD to allow an agent to efficiently replay all or part of its past experience, online in the sequence that they appear with, either in each step or sparsely according to the usual {\lambda} parameter. In this new method that we call True Online TD-Replan({\lambda}), the {\lambda} parameter plays a new role in specifying the density of the replay process in addition to the usual role of specifying the depth of the target's updates. We demonstrate that, for problems that benefit from experience replay, our new method outperforms true online TD({\lambda}), albeit quadratic in complexity due to its replay capabilities. In addition, we demonstrate that our method outperforms other methods with similar quadratic complexity such as Dyna Planning and TD({\lambda})-Replan algorithms. We test our method on two benchmarking environments, a random walk problem that uses simple binary features and a myoelectric control domain that uses both simple sEMG features and deeply extracted features to showcase its capabilities.