An Empirical Study of Deep Reinforcement Learning in Continuing Tasks

TL;DR

This paper empirically investigates deep reinforcement learning algorithms in continuing tasks, highlighting their behaviors, challenges, and the effectiveness of reward-centering methods across various algorithms and large-scale environments.

Contribution

It provides the first comprehensive empirical analysis of deep RL in continuing tasks and extends reward-centering techniques to multiple algorithms and larger environments.

Findings

Reward-centering improves performance across algorithms.

Deep RL algorithms behave differently in continuing tasks.

Reward-centering outperforms other methods in large-scale environments.

Abstract

In reinforcement learning (RL), continuing tasks refer to tasks where the agent-environment interaction is ongoing and can not be broken down into episodes. These tasks are suitable when environment resets are unavailable, agent-controlled, or predefined but where all rewards-including those beyond resets-are critical. These scenarios frequently occur in real-world applications and can not be modeled by episodic tasks. While modern deep RL algorithms have been extensively studied and well understood in episodic tasks, their behavior in continuing tasks remains underexplored. To address this gap, we provide an empirical study of several well-known deep RL algorithms using a suite of continuing task testbeds based on Mujoco and Atari environments, highlighting several key insights concerning continuing tasks. Using these testbeds, we also investigate the effectiveness of a method for improving temporal-difference-based RL algorithms in continuing tasks by centering rewards, as introduced by Naik et al. (2024). While their work primarily focused on this method in conjunction with Q-learning, our results extend their findings by demonstrating that this method is effective across a broader range of algorithms, scales to larger tasks, and outperforms two other reward-centering approaches.