ACSAC: Adaptive Chunk Size Actor-Critic with Causal Transformer Q-Network

TL;DR

ACSAC introduces an adaptive chunk size actor-critic method using a causal Transformer critic to dynamically select action chunk lengths, improving performance on long-horizon, sparse-reward tasks without manual tuning.

Contribution

It proposes ACSAC, a novel reinforcement learning algorithm that adaptively chooses action chunk sizes with a Transformer critic, addressing fixed chunk size limitations.

Findings

Achieves state-of-the-art results on long-horizon manipulation tasks.

Demonstrates effective adaptive chunk size selection without task-specific tuning.

Proves the Bellman operator for ACSAC is a contraction with a unique fixed point.

Abstract

Long-horizon, sparse-reward tasks pose a fundamental challenge for reinforcement learning, since single-step TD learning suffers from bootstrapping error accumulation across successive Bellman updates. Actor-critic methods with action chunking address this by operating over temporally extended actions, which reduce the effective horizon, enable fast value backups, and support temporally consistent exploration. However, existing methods rely on a fixed chunk size and therefore cannot adaptively balance reactivity against temporal consistency. A large fixed chunk size reduces responsiveness to new observations, while a small one produces incoherent motions, forcing task-specific tuning of the chunk size. To address this limitation, we propose Adaptive Chunk Size Actor-Critic (ACSAC). ACSAC leverages a causal Transformer critic to evaluate expected returns for action chunks of different sizes. At each chunk boundary, it adaptively selects the chunk size that maximizes the expected return, supporting flexible, state-dependent chunk sizes without task-specific tuning. We prove that the ACSAC Bellman operator is a contraction whose unique fixed point is the action-value function of the adaptive policy. Experiments on OGBench demonstrate that ACSAC achieves state-of-the-art performance on long-horizon, sparse-reward manipulation tasks across both offline RL and offline-to-online RL settings.