Extending Differential Temporal Difference Methods for Episodic Problems

TL;DR

This paper extends differential TD methods to episodic reinforcement learning problems by generalizing reward centering, maintaining policy orderings, and improving sample efficiency.

Contribution

It introduces a theoretical generalization of differential TD for episodic tasks and empirically demonstrates improved sample efficiency across algorithms.

Findings

Reward centering can improve sample efficiency in episodic problems.

The proposed generalization maintains policy ordering with termination.

Differential versions of streaming RL algorithms inherit theoretical guarantees.

Abstract

Differential temporal difference (TD) methods are value-based reinforcement learning algorithms that have been proposed for infinite-horizon problems. They rely on reward centering, where each reward is centered by the average reward. This keeps the return bounded and removes a value function's state-independent offset. However, reward centering can alter the optimal policy in episodic problems, limiting its applicability. Motivated by recent works that emphasize the role of normalization in streaming deep reinforcement learning, we study reward centering in episodic problems and propose a generalization of differential TD. We prove that this generalization maintains the ordering of policies in the presence of termination, and thus extends differential TD to episodic problems. We show equivalence with a form of linear TD, thereby inheriting theoretical guarantees that have been shown for those algorithms. We then extend several streaming reinforcement learning algorithms to their differential counterparts. Across a range of base algorithms and environments, we empirically validate that reward centering can improve sample efficiency in episodic problems.