On the Synthesis of Bellman Inequalities for Data-Driven Optimal Control

TL;DR

This paper develops methods to generate Bellman inequalities from limited data for linear systems, enabling offline constraint generation and cost reconstruction in data-driven optimal control without extensive exploration.

Contribution

It introduces a novel approach to synthesize Bellman inequalities using small datasets, facilitating offline control design and cost estimation for unknown linear systems.

Findings

Efficient offline generation of Bellman constraints from limited data

Reconstruction of unknown stage costs from data

Insights into estimating Bellman expectations in stochastic systems

Abstract

In the context of the linear programming (LP) approach to data-driven control, one assumes that the dynamical system is unknown but can be observed indirectly through data on its evolution. Both theoretical and empirical evidence suggest that a desired suboptimality gap is often only achieved with massive exploration of the state-space. In case of linear systems, we discuss how a relatively small but sufficiently rich dataset can be exploited to generate new constraints offline and without observing the corresponding transitions. Moreover, we show how to reconstruct the associated unknown stage-costs and, when the system is stochastic, we offer insights on the related problem of estimating the expected value in the Bellman operator without re-initializing the dynamics in the same state-input pairs.