SEAR: Sample Efficient Action Chunking Reinforcement Learning

TL;DR

SEAR is an off-policy online reinforcement learning algorithm that leverages action chunking with a receding horizon to improve exploration and performance in long-horizon tasks, outperforming existing methods.

Contribution

Introduces SEAR, a novel off-policy online RL algorithm that effectively utilizes action chunking with a receding horizon for better long-horizon learning.

Findings

SEAR outperforms state-of-the-art online RL methods on Metaworld.

SEAR effectively handles chunk sizes up to 20.

The method improves exploration and value estimation in long-horizon tasks.

Abstract

Action chunking can improve exploration and value estimation in long horizon reinforcement learning, but makes learning substantially harder since the critic must evaluate action sequences rather than single actions, greatly increasing approximation and data efficiency challenges. As a result, existing action chunking methods, primarily designed for the offline and offline-to-online settings, have not achieved strong performance in purely online reinforcement learning. We introduce SEAR, an off policy online reinforcement learning algorithm for action chunking. It exploits the temporal structure of action chunks and operates with a receding horizon, effectively combining the benefits of small and large chunk sizes. SEAR outperforms state of the art online reinforcement learning methods on Metaworld, training with chunk sizes up to 20.