Fast TRAC: A Parameter-Free Optimizer for Lifelong Reinforcement Learning

TL;DR

This paper introduces TRAC, a parameter-free optimizer for lifelong reinforcement learning that adapts rapidly to new tasks without hyperparameter tuning, effectively mitigating loss of plasticity across diverse environments.

Contribution

The paper presents TRAC, a novel parameter-free optimizer based on online convex optimization theory, specifically designed for lifelong RL to handle nonconvex, nonstationary problems without prior tuning.

Findings

TRAC effectively mitigates loss of plasticity in RL agents.

TRAC adapts quickly to challenging distribution shifts.

Experimental results show TRAC outperforms traditional methods.

Abstract

A key challenge in lifelong reinforcement learning (RL) is the loss of plasticity, where previous learning progress hinders an agent's adaptation to new tasks. While regularization and resetting can help, they require precise hyperparameter selection at the outset and environment-dependent adjustments. Building on the principled theory of online convex optimization, we present a parameter-free optimizer for lifelong RL, called TRAC, which requires no tuning or prior knowledge about the distribution shifts. Extensive experiments on Procgen, Atari, and Gym Control environments show that TRAC works surprisingly well-mitigating loss of plasticity and rapidly adapting to challenging distribution shifts-despite the underlying optimization problem being nonconvex and nonstationary.