Addressing Action Oscillations through Learning Policy Inertia

TL;DR

This paper introduces Policy Inertia Controller (PIC) to reduce action oscillations in deep reinforcement learning, improving safety and user experience without sacrificing performance across various tasks.

Contribution

The paper proposes a novel plug-in framework, PIC, and a training algorithm, Nested Policy Iteration, to balance policy optimality and smoothness in DRL.

Findings

Significant reduction in action oscillations in autonomous driving and Atari games.

Almost no performance loss compared to baseline algorithms.

Demonstrates effectiveness across multiple challenging tasks.

Abstract

Deep reinforcement learning (DRL) algorithms have been demonstrated to be effective in a wide range of challenging decision making and control tasks. However, these methods typically suffer from severe action oscillations in particular in discrete action setting, which means that agents select different actions within consecutive steps even though states only slightly differ. This issue is often neglected since the policy is usually evaluated by its cumulative rewards only. Action oscillation strongly affects the user experience and can even cause serious potential security menace especially in real-world domains with the main concern of safety, such as autonomous driving. To this end, we introduce Policy Inertia Controller (PIC) which serves as a generic plug-in framework to off-the-shelf DRL algorithms, to enables adaptive trade-off between the optimality and smoothness of the learned policy in a formal way. We propose Nested Policy Iteration as a general training algorithm for PIC-augmented policy which ensures monotonically non-decreasing updates under some mild conditions. Further, we derive a practical DRL algorithm, namely Nested Soft Actor-Critic. Experiments on a collection of autonomous driving tasks and several Atari games suggest that our approach demonstrates substantial oscillation reduction in comparison to a range of commonly adopted baselines with almost no performance degradation.