Logarithmic Regret for Nonlinear Control

TL;DR

This paper establishes logarithmic regret bounds for controlling unknown nonlinear dynamical systems, demonstrating that fast learning is possible under certain conditions, with implications for high-stakes applications.

Contribution

It provides the first regret bounds for nonlinear systems with nonlinear parameter dependence and shows conditions for achieving logarithmic regret in control tasks.

Findings

Logarithmic regret achievable with persistently exciting policies.

Regret bounds grow with the square root of interactions without persistent excitation.

Validation of theoretical results through simulation on a simple system.

Abstract

We address the problem of learning to control an unknown nonlinear dynamical system through sequential interactions. Motivated by high-stakes applications in which mistakes can be catastrophic, such as robotics and healthcare, we study situations where it is possible for fast sequential learning to occur. Fast sequential learning is characterized by the ability of the learning agent to incur logarithmic regret relative to a fully-informed baseline. We demonstrate that fast sequential learning is achievable in a diverse class of continuous control problems where the system dynamics depend smoothly on unknown parameters, provided the optimal control policy is persistently exciting. Additionally, we derive a regret bound which grows with the square root of the number of interactions for cases where the optimal policy is not persistently exciting. Our results provide the first regret bounds for controlling nonlinear dynamical systems depending nonlinearly on unknown parameters. We validate the trends our theory predicts in simulation on a simple dynamical system.