Deep Reinforcement Learning with Embedded LQR Controllers

TL;DR

This paper explores integrating Linear Quadratic Regulator (LQR) control with reinforcement learning to improve control performance in reaching tasks, addressing issues like chattering and enhancing generalization.

Contribution

It introduces methods that embed LQR control into reinforcement learning algorithms, enabling better performance and robustness in control tasks.

Findings

Adding LQR control improves RL performance

Embedding LQR into discrete actions is particularly effective

LQR integration helps mitigate chattering in reaching tasks

Abstract

Reinforcement learning is a model-free optimal control method that optimizes a control policy through direct interaction with the environment. For reaching tasks that end in regulation, popular discrete-action methods are not well suited due to chattering in the goal state. We compare three different ways to solve this problem through combining reinforcement learning with classical LQR control. In particular, we introduce a method that integrates LQR control into the action set, allowing generalization and avoiding fixing the computed control in the replay memory if it is based on learned dynamics. We also embed LQR control into a continuous-action method. In all cases, we show that adding LQR control can improve performance, although the effect is more profound if it can be used to augment a discrete action set.