Online Learning of Feasible Strategies in Unknown Environments

TL;DR

This paper introduces an online saddle point controller for environments with time-varying convex constraints and costs, enabling agents to learn feasible and near-optimal strategies with bounded or sublinear regret and constraint violations.

Contribution

It proposes a novel online control method using saddle point dynamics that guarantees bounded or sublinear growth of regret and constraint violations in unknown, dynamic environments.

Findings

Controller achieves bounded fit and regret.

Method effectively manages time-varying constraints and costs.

Numerical experiments validate the approach in a shepherding scenario.

Abstract

Define an environment as a set of convex constraint functions that vary arbitrarily over time and consider a cost function that is also convex and arbitrarily varying. Agents that operate in this environment intend to select actions that are feasible for all times while minimizing the cost's time average. Such action is said optimal and can be computed offline if the cost and the environment are known a priori. An online policy is one that depends causally on the cost and the environment. To compare online policies to the optimal offline action define the fit of a trajectory as a vector that integrates the constraint violations over time and its regret as the cost difference with the optimal action accumulated over time. Fit measures the extent to which an online policy succeeds in learning feasible actions while regret measures its success in learning optimal actions. This paper proposes the use of online policies computed from a saddle point controller which are shown to have fit and regret that are either bounded or grow at a sublinear rate. These properties provide an indication that the controller finds trajectories that are feasible and optimal in a relaxed sense. Concepts are illustrated throughout with the problem of a shepherd that wants to stay close to all sheep in a herd. Numerical experiments show that the saddle point controller allows the shepherd to do so.