Improving the Efficiency of Off-Policy Reinforcement Learning by Accounting for Past Decisions

TL;DR

This paper introduces a new multistep operator for off-policy reinforcement learning that accounts for past decisions, providing convergence guarantees and improving learning efficiency over existing methods.

Contribution

It proposes a novel past-dependent trace operator for off-policy learning, with proven convergence and applicability to existing algorithms, enhancing credit assignment and learning speed.

Findings

Proves convergence of the new operator for policy evaluation and control.

Establishes convergence guarantees for algorithms like Truncated IS and Retrace.

Guides development of algorithms considering multiple past decisions.

Abstract

Off-policy learning from multistep returns is crucial for sample-efficient reinforcement learning, particularly in the experience replay setting now commonly used with deep neural networks. Classically, off-policy estimation bias is corrected in a per-decision manner: past temporal-difference errors are re-weighted by the instantaneous Importance Sampling (IS) ratio (via eligibility traces) after each action. Many important off-policy algorithms such as Tree Backup and Retrace rely on this mechanism along with differing protocols for truncating ("cutting") the ratios ("traces") to counteract the excessive variance of the IS estimator. Unfortunately, cutting traces on a per-decision basis is not necessarily efficient; once a trace has been cut according to local information, the effect cannot be reversed later, potentially resulting in the premature truncation of estimated returns and slower learning. In the interest of motivating efficient off-policy algorithms, we propose a multistep operator that permits arbitrary past-dependent traces. We prove that our operator is convergent for policy evaluation, and for optimal control when targeting greedy-in-the-limit policies. Our theorems establish the first convergence guarantees for many existing algorithms including Truncated IS, Non-Markov Retrace, and history-dependent TD($\lambda$). Our theoretical results also provide guidance for the development of new algorithms that jointly consider multiple past decisions for better credit assignment and faster learning.