# Learning the optimal state-feedback via supervised imitation learning

## TL;DR

This paper demonstrates how supervised imitation learning with deep neural networks can accurately approximate optimal state-feedback controllers for a quadcopter model, achieving near-optimal performance with high precision.

## Contribution

The study refines previous imitation learning methods by introducing a robust training pipeline, including the use of softplus activation, to accurately learn optimal control policies for nonlinear systems.

## Key findings

- Deep neural networks can approximate optimal state-feedback with less than 1% error.
- Two-layer networks achieve near-optimal control performance.
- Improvements in mean absolute error do not always lead to better control policies.

## Abstract

Imitation learning is a control design paradigm that seeks to learn a control policy reproducing demonstrations from expert agents. By substituting expert demonstrations for optimal behaviours, the same paradigm leads to the design of control policies closely approximating the optimal state-feedback. This approach requires training a machine learning algorithm (in our case deep neural networks) directly on state-control pairs originating from optimal trajectories. We have shown in previous work that, when restricted to low-dimensional state and control spaces, this approach is very successful in several deterministic, non-linear problems in continuous-time. In this work, we refine our previous studies using as a test case a simple quadcopter model with quadratic and time-optimal objective functions. We describe in detail the best learning pipeline we have developed, that is able to approximate via deep neural networks the state-feedback map to a very high accuracy. We introduce the use of the softplus activation function in the hidden units of neural networks showing that it results in a smoother control profile whilst retaining the benefits of rectifiers. We show how to evaluate the optimality of the trained state-feedback, and find that already with two layers the objective function reached and its optimal value differ by less than one percent. We later consider also an additional metric linked to the system asymptotic behaviour - time taken to converge to the policy's fixed point. With respect to these metrics, we show that improvements in the mean absolute error do not necessarily correspond to better policies.