Benchmarking Smoothness and Reducing High-Frequency Oscillations in Continuous Control Policies

TL;DR

This paper benchmarks various methods to reduce high-frequency oscillations in deep reinforcement learning policies, proposing hybrid approaches that improve control smoothness with minimal performance loss, especially in real-world robotics tasks.

Contribution

It provides a comprehensive comparison of regularization and architectural methods for smoothing RL policies and introduces hybrid techniques that outperform existing approaches.

Findings

Hybrid methods outperform individual techniques in smoothness.

Control smoothness improved by 26.8% with minimal performance degradation.

Benchmarks include real-world robotics deployment.

Abstract

Reinforcement learning (RL) policies are prone to high-frequency oscillations, especially undesirable when deploying to hardware in the real-world. In this paper, we identify, categorize, and compare methods from the literature that aim to mitigate high-frequency oscillations in deep RL. We define two broad classes: loss regularization and architectural methods. At their core, these methods incentivize learning a smooth mapping, such that nearby states in the input space produce nearby actions in the output space. We present benchmarks in terms of policy performance and control smoothness on traditional RL environments from the Gymnasium and a complex manipulation task, as well as three robotics locomotion tasks that include deployment and evaluation with real-world hardware. Finally, we also propose hybrid methods that combine elements from both loss regularization and architectural methods. We find that the best-performing hybrid outperforms other methods, and improves control smoothness by 26.8% over the baseline, with a worst-case performance degradation of just 2.8%.