Variable Time Step Reinforcement Learning for Robotic Applications

TL;DR

This paper introduces VTS-RL, an adaptive control frequency method for reinforcement learning in robotics, improving efficiency and performance by dynamically adjusting action timing.

Contribution

The paper presents MOSEAC, a novel algorithm for VTS-RL, validated through theoretical analysis and experiments, demonstrating faster convergence and energy savings.

Findings

Faster convergence compared to fixed-frequency RL.

Reduced energy consumption in robotic tasks.

Improved training outcomes with adaptive control frequencies.

Abstract

Traditional reinforcement learning (RL) generates discrete control policies, assigning one action per cycle. These policies are usually implemented as in a fixed-frequency control loop. This rigidity presents challenges as optimal control frequency is task-dependent; suboptimal frequencies increase computational demands and reduce exploration efficiency. Variable Time Step Reinforcement Learning (VTS-RL) addresses these issues with adaptive control frequencies, executing actions only when necessary, thus reducing computational load and extending the action space to include action durations. In this paper we introduce the Multi-Objective Soft Elastic Actor-Critic (MOSEAC) method to perform VTS-RL, validating it through theoretical analysis and experimentation in simulation and on real robots. Results show faster convergence, better training results, and reduced energy consumption with respect to other variable- or fixed-frequency approaches.