Continuous Control with Coarse-to-fine Reinforcement Learning

TL;DR

CRL introduces a coarse-to-fine approach for reinforcement learning that enhances sample efficiency and stability in continuous control tasks by iteratively discretizing action spaces and selecting optimal intervals.

Contribution

The paper proposes the CRL framework and the CQN algorithm, enabling stable, sample-efficient learning for fine-grained continuous control through a novel discretization strategy.

Findings

CQN outperforms RL and behavior cloning baselines on RLBench tasks.

CQN learns real-world manipulation tasks within minutes.

CRL improves sample efficiency and robustness in continuous control.

Abstract

Despite recent advances in improving the sample-efficiency of reinforcement learning (RL) algorithms, designing an RL algorithm that can be practically deployed in real-world environments remains a challenge. In this paper, we present Coarse-to-fine Reinforcement Learning (CRL), a framework that trains RL agents to zoom-into a continuous action space in a coarse-to-fine manner, enabling the use of stable, sample-efficient value-based RL algorithms for fine-grained continuous control tasks. Our key idea is to train agents that output actions by iterating the procedure of (i) discretizing the continuous action space into multiple intervals and (ii) selecting the interval with the highest Q-value to further discretize at the next level. We then introduce a concrete, value-based algorithm within the CRL framework called Coarse-to-fine Q-Network (CQN). Our experiments demonstrate that CQN significantly outperforms RL and behavior cloning baselines on 20 sparsely-rewarded RLBench manipulation tasks with a modest number of environment interactions and expert demonstrations. We also show that CQN robustly learns to solve real-world manipulation tasks within a few minutes of online training.