Actor-Critic with variable time discretization via sustained actions

TL;DR

This paper introduces SusACER, an off-policy reinforcement learning algorithm that adaptively transitions from sparse to fine time discretization, improving robotic control performance across various environments.

Contribution

It proposes a novel RL method that dynamically adjusts time discretization, combining the benefits of different control granularities for better robotic task performance.

Findings

SusACER outperforms state-of-the-art methods in robotic control environments.

Adaptive time discretization improves learning efficiency and control quality.

The approach is effective across multiple simulated robotic tasks.

Abstract

Reinforcement learning (RL) methods work in discrete time. In order to apply RL to inherently continuous problems like robotic control, a specific time discretization needs to be defined. This is a choice between sparse time control, which may be easier to train, and finer time control, which may allow for better ultimate performance. In this work, we propose SusACER, an off-policy RL algorithm that combines the advantages of different time discretization settings. Initially, it operates with sparse time discretization and gradually switches to a fine one. We analyze the effects of the changing time discretization in robotic control environments: Ant, HalfCheetah, Hopper, and Walker2D. In all cases our proposed algorithm outperforms state of the art.