Adaptive Control of Positive Systems with Application to Learning SSP

TL;DR

This paper introduces an adaptive, data-driven control method for positive linear systems solving infinite-horizon optimal control problems, with applications to learning stochastic shortest path problems, emphasizing efficiency, stability, and robustness.

Contribution

It presents a novel adaptive controller derived from a model-free Bellman equation that scales efficiently and guarantees stability in positive systems for SSP problems.

Findings

Efficient online implementation of the controller.

Guarantees of stability and robustness to unmodeled dynamics.

Successful application to learning stochastic shortest path problems.

Abstract

An adaptive controller is proposed and analyzed for the class of infinite-horizon optimal control problems in positive linear systems presented in (Ohlin et al., 2024b). This controller is derived from the solution of a "data-driven algebraic equation" constructed using the model-free Bellman equation from Q-learning. The equation is driven by data correlation matrices that do not scale with the number of data points, enabling efficient online implementation. Consequently, a sufficient condition guaranteeing stability and robustness to unmodeled dynamics is established. The derived results also provide a quantitative characterization of the interplay between excitation level and robustness to unmodeled dynamics. The class of optimal control problems considered here is equivalent to Stochastic Shortest Path (SSP) problems, allowing for a performance comparison between the proposed adaptive policy and model-free algorithms for learning the stochastic shortest path, as demonstrated in the numerical experiment.