Gradient-based Regularization for Action Smoothness in Robotic Control with Reinforcement Learning

TL;DR

This paper introduces Grad-CAPS, a gradient-based regularization method that enhances action smoothness in reinforcement learning for robotics, reducing jerky movements and improving policy performance across diverse environments.

Contribution

The paper proposes Grad-CAPS, a novel regularization technique that modifies CAPS by focusing on gradient differences, improving action smoothness and adaptability in reinforcement learning for robotic control.

Findings

Grad-CAPS reduces jerky actions effectively.

It improves policy performance across multiple environments.

Maintains comparable smoothness to existing methods.

Abstract

Deep Reinforcement Learning (DRL) has achieved remarkable success, ranging from complex computer games to real-world applications, showing the potential for intelligent agents capable of learning in dynamic environments. However, its application in real-world scenarios presents challenges, including the jerky problem, in which jerky trajectories not only compromise system safety but also increase power consumption and shorten the service life of robotic and autonomous systems. To address jerky actions, a method called conditioning for action policy smoothness (CAPS) was proposed by adding regularization terms to reduce the action changes. This paper further proposes a novel method, named Gradient-based CAPS (Grad-CAPS), that modifies CAPS by reducing the difference in the gradient of action and then uses displacement normalization to enable the agent to adapt to invariant action scales. Consequently, our method effectively reduces zigzagging action sequences while enhancing policy expressiveness and the adaptability of our method across diverse scenarios and environments. In the experiments, we integrated Grad-CAPS with different reinforcement learning algorithms and evaluated its performance on various robotic-related tasks in DeepMind Control Suite and OpenAI Gym environments. The results demonstrate that Grad-CAPS effectively improves performance while maintaining a comparable level of smoothness compared to CAPS and Vanilla agents.