Adaptive Online Non-stochastic Control

TL;DR

This paper develops adaptive algorithms for non-stochastic control that achieve regret bounds proportional to environment difficulty, using a novel FTRL approach with cost-based regularizers and new analysis techniques.

Contribution

It introduces a new adaptive FTRL-based framework for NSC with state, providing sub-linear, data-dependent regret bounds that improve with easier environments.

Findings

Achieved sub-linear, adaptive policy regret bounds.

Developed disturbance action controllers with improved performance.

Provided new analysis tools for NSC with memory effects.

Abstract

We tackle the problem of Non-stochastic Control (NSC) with the aim of obtaining algorithms whose policy regret is proportional to the difficulty of the controlled environment. Namely, we tailor the Follow The Regularized Leader (FTRL) framework to dynamical systems by using regularizers that are proportional to the actual witnessed costs. The main challenge arises from using the proposed adaptive regularizers in the presence of a state, or equivalently, a memory, which couples the effect of the online decisions and requires new tools for bounding the regret. Via new analysis techniques for NSC and FTRL integration, we obtain novel disturbance action controllers (DAC) with sub-linear data adaptive policy regret bounds that shrink when the trajectory of costs has small gradients, while staying sub-linear even in the worst case.