Robustness to Model Approximation, Model Learning From Data, and Sample Complexity in Wasserstein Regular MDPs

TL;DR

This paper analyzes how Wasserstein model approximation affects the robustness of stochastic optimal control policies, providing bounds on performance loss and sample complexity in empirical model learning scenarios.

Contribution

It establishes robustness bounds for control policies under Wasserstein approximation and connects these to empirical learning and disturbance estimation.

Findings

Performance loss is bounded by Wasserstein-1 distance between models.

Sample complexity bounds are derived for empirical model learning.

Robustness results apply to both discounted and average-cost criteria.

Abstract

The paper studies the robustness properties of discrete-time stochastic optimal control under Wasserstein model approximation for both discounted-cost and average-cost criteria. Specifically, we study the performance loss when applying an optimal policy designed for an approximate model to the true dynamics compared with the optimal cost for the true model under the sup-norm-induced metric, and relate it to the Wasserstein-1 distance between the approximate and true transition kernels. A primary motivation of this analysis is empirical model learning, as well as empirical noise distribution learning, where Wasserstein convergence holds under mild conditions but stronger convergence criteria, such as total variation, may not. We discuss applications of the results to the disturbance estimation problem, where sample complexity bounds are given, and also to a general empirical model learning approach, obtained under either Markov or i.i.d. learning settings.