Reinforcement Learning-based Control via Y-wise Affine Neural Networks (YANNs)

TL;DR

This paper introduces a reinforcement learning algorithm utilizing Y-wise Affine Neural Networks (YANNs) for interpretable control, enabling initial linear optimal control confidence and online adaptation to nonlinear systems.

Contribution

The novel integration of YANNs with RL provides a method for initializing policies with linear control solutions and extending to nonlinear systems through online training.

Findings

YANN-RL outperforms deep deterministic policy gradient in safety-critical systems.

YANNs can exactly represent piecewise affine functions on polytopic subdomains.

The approach effectively combines explicit linear control solutions with online nonlinear learning.

Abstract

This work presents a novel reinforcement learning (RL) algorithm based on Y-wise Affine Neural Networks (YANNs). YANNs provide an interpretable neural network which can exactly represent known piecewise affine functions of arbitrary input and output dimensions defined on any amount of polytopic subdomains. One representative application of YANNs is to reformulate explicit solutions of multi-parametric linear model predictive control. Built on this, we propose the use of YANNs to initialize RL actor and critic networks, which enables the resulting YANN-RL control algorithm to start with the confidence of linear optimal control. The YANN-actor is initialized by representing the multi-parametric control solutions obtained via offline computation using an approximated linear system model. The YANN-critic represents the explicit form of the state-action value function for the linear system and the reward function as the objective in an optimal control problem (OCP). Additional network layers are injected to extend YANNs for nonlinear expressions, which can be trained online by directly interacting with the true complex nonlinear system. In this way, both the policy and state-value functions exactly represent a linear OCP initially and are able to eventually learn the solution of a general nonlinear OCP. Continuous policy improvement is also implemented to provide heuristic confidence that the linear OCP solution serves as an effective lower bound to the performance of RL policy. The YANN-RL algorithm is demonstrated on a clipped pendulum and a safety-critical chemical-reactive system. Our results show that YANN-RL significantly outperforms the modern RL algorithm using deep deterministic policy gradient, especially when considering safety constraints.