Hysteresis-Based RL: Robustifying Reinforcement Learning-based Control Policies via Hybrid Control

TL;DR

This paper introduces HyRL, a hybrid reinforcement learning algorithm that enhances robustness of control policies by incorporating hysteresis switching and dual learning stages, addressing limitations of PPO and DQN in complex systems.

Contribution

The paper proposes HyRL, a novel hybrid RL method that improves robustness of control policies through hysteresis switching and two-stage learning, demonstrated on challenging control problems.

Findings

HyRL outperforms PPO and DQN in robustness on tested problems.

Hysteresis switching enhances stability of learned policies.

Two-stage learning improves policy robustness and performance.

Abstract

Reinforcement learning (RL) is a promising approach for deriving control policies for complex systems. As we show in two control problems, the derived policies from using the Proximal Policy Optimization (PPO) and Deep Q-Network (DQN) algorithms may lack robustness guarantees. Motivated by these issues, we propose a new hybrid algorithm, which we call Hysteresis-Based RL (HyRL), augmenting an existing RL algorithm with hysteresis switching and two stages of learning. We illustrate its properties in two examples for which PPO and DQN fail.