Offline Supervised Learning V.S. Online Direct Policy Optimization: A Comparative Study and A Unified Training Paradigm for Neural Network-Based Optimal Feedback Control

TL;DR

This paper compares offline supervised learning and online direct policy optimization for neural network-based feedback controllers, highlighting their strengths and weaknesses, and proposes a unified pre-train and fine-tune paradigm to enhance performance and robustness.

Contribution

It provides a comprehensive comparison of two prevalent control training methods and introduces a unified training paradigm to improve neural network-based optimal feedback control.

Findings

Offline supervised learning outperforms in optimality and training time.

Direct policy optimization faces challenges with complex dynamics.

The proposed unified paradigm significantly enhances control performance and robustness.

Abstract

This work is concerned with solving neural network-based feedback controllers efficiently for optimal control problems. We first conduct a comparative study of two prevalent approaches: offline supervised learning and online direct policy optimization. Albeit the training part of the supervised learning approach is relatively easy, the success of the method heavily depends on the optimal control dataset generated by open-loop optimal control solvers. In contrast, direct policy optimization turns the optimal control problem into an optimization problem directly without any requirement of pre-computing, but the dynamics-related objective can be hard to optimize when the problem is complicated. Our results underscore the superiority of offline supervised learning in terms of both optimality and training time. To overcome the main challenges, dataset and optimization, in the two approaches respectively, we complement them and propose the Pre-train and Fine-tune strategy as a unified training paradigm for optimal feedback control, which further improves the performance and robustness significantly. Our code is accessible at https://github.com/yzhao98/DeepOptimalControl.