Enhancing Control Policy Smoothness by Aligning Actions with Predictions from Preceding States

TL;DR

This paper introduces ASAP, a novel reinforcement learning method that enhances control policy smoothness by aligning actions with predicted outcomes from previous states, leading to more stable and effective control in dynamic environments.

Contribution

The paper proposes a new loss-based approach using transition-induced similar states to better reflect system dynamics and improve action smoothness in reinforcement learning policies.

Findings

ASAP achieves smoother control in simulated environments.

ASAP outperforms existing methods in policy performance.

The approach effectively reduces high-frequency oscillations.

Abstract

Deep reinforcement learning has proven to be a powerful approach to solving control tasks, but its characteristic high-frequency oscillations make it difficult to apply in real-world environments. While prior methods have addressed action oscillations via architectural or loss-based methods, the latter typically depend on heuristic or synthetic definitions of state similarity to promote action consistency, which often fail to accurately reflect the underlying system dynamics. In this paper, we propose a novel loss-based method by introducing a transition-induced similar state. The transition-induced similar state is defined as the distribution of next states transitioned from the previous state. Since it utilizes only environmental feedback and actually collected data, it better captures system dynamics. Building upon this foundation, we introduce Action Smoothing by Aligning Actions with Predictions from Preceding States (ASAP), an action smoothing method that effectively mitigates action oscillations. ASAP enforces action smoothness by aligning the actions with those taken in transition-induced similar states and by penalizing second-order differences to suppress high-frequency oscillations. Experiments in Gymnasium and Isaac-Lab environments demonstrate that ASAP yields smoother control and improved policy performance over existing methods.