Change Point Detection Approach for Online Control of Unknown Time Varying Dynamical Systems

TL;DR

This paper introduces a change point detection method for online control of unknown, time-varying dynamical systems, achieving sub-linear regret and outperforming existing approaches in adaptive control scenarios.

Contribution

The paper presents the first regret guarantee for unknown time-varying systems using change point detection, applicable to general classes of systems with sub-linear changes.

Findings

Achieves a regret of $ ext{Gamma}_T^{1/5} T^{4/5}$ for systems with $ ext{Gamma}_T$ changes.

Demonstrates superior performance over restart and standard online learning methods.

Provides the first regret bounds based on the number of system changes.

Abstract

We propose a novel change point detection approach for online learning control with full information feedback (state, disturbance, and cost feedback) for unknown time-varying dynamical systems. We show that our algorithm can achieve a sub-linear regret with respect to the class of Disturbance Action Control (DAC) policies, which are a widely studied class of policies for online control of dynamical systems, for any sub-linear number of changes and very general class of systems: (i) matched disturbance system with general convex cost functions, (ii) general system with linear cost functions. Specifically, a (dynamic) regret of $\Gamma_T^{1/5}T^{4/5}$ can be achieved for these class of systems, where $\Gamma_T$ is the number of changes of the underlying system and $T$ is the duration of the control episode. That is, the change point detection approach achieves a sub-linear regret for any sub-linear number of changes, which other previous algorithms such as in \cite{minasyan2021online} cannot. Numerically, we demonstrate that the change point detection approach is superior to a standard restart approach \cite{minasyan2021online} and to standard online learning approaches for time-invariant dynamical systems. Our work presents the first regret guarantee for unknown time-varying dynamical systems in terms of a stronger notion of variability like the number of changes in the underlying system. The extension of our work to state and output feedback controllers is a subject of future work.