Extended Radial Basis Function Controller for Reinforcement Learning

TL;DR

This paper introduces a hybrid reinforcement learning controller that combines a model-based linear controller with a flexible policy, ensuring stability near an operating point while maintaining universal approximation capabilities.

Contribution

It proposes a novel hybrid controller that interpolates between linear and non-linear policies, with stability guarantees and applicability to both model-based and model-free RL frameworks.

Findings

Demonstrates stability and robustness in simulations

Maintains stability around the operating point

Combines control theory with reinforcement learning

Abstract

There have been attempts in reinforcement learning to exploit a priori knowledge about the structure of the system. This paper proposes a hybrid reinforcement learning controller which dynamically interpolates a model-based linear controller and an arbitrary differentiable policy. The linear controller is designed based on local linearised model knowledge, and stabilises the system in a neighbourhood about an operating point. The coefficients of interpolation between the two controllers are determined by a scaled distance function measuring the distance between the current state and the operating point. The overall hybrid controller is proven to maintain the stability guarantee around the neighborhood of the operating point and still possess the universal function approximation property of the arbitrary non-linear policy. Learning has been done on both model-based (PILCO) and model-free (DDPG) frameworks. Simulation experiments performed in OpenAI gym demonstrate stability and robustness of the proposed hybrid controller. This paper thus introduces a principled method allowing for the direct importing of control methodology into reinforcement learning.