Active Reinforcement Learning Strategies for Offline Policy Improvement

TL;DR

This paper introduces an active reinforcement learning approach that efficiently collects minimal new data to improve policies, significantly reducing online interactions in various complex environments.

Contribution

It presents the first active learning method tailored for offline reinforcement learning, effectively reducing environment interactions by up to 75%.

Findings

Reduces online environment interactions by up to 75%.

Effective across diverse environments like MuJoCo, Maze2d, and CARLA.

First to address active learning in sequential decision-making RL.

Abstract

Learning agents that excel at sequential decision-making tasks must continuously resolve the problem of exploration and exploitation for optimal learning. However, such interactions with the environment online might be prohibitively expensive and may involve some constraints, such as a limited budget for agent-environment interactions and restricted exploration in certain regions of the state space. Examples include selecting candidates for medical trials and training agents in complex navigation environments. This problem necessitates the study of active reinforcement learning strategies that collect minimal additional experience trajectories by reusing existing offline data previously collected by some unknown behavior policy. In this work, we propose an active reinforcement learning method capable of collecting trajectories that can augment existing offline data. With extensive experimentation, we demonstrate that our proposed method reduces additional online interaction with the environment by up to 75% over competitive baselines across various continuous control environments such as Gym-MuJoCo locomotion environments as well as Maze2d, AntMaze, CARLA and IsaacSimGo1. To the best of our knowledge, this is the first work that addresses the active learning problem in the context of sequential decision-making and reinforcement learning.