Certifiably Robust Policies for Uncertain Parametric Environments

TL;DR

This paper introduces a data-driven framework for creating policies that are provably robust across uncertain environments by combining parametric MDPs with scenario optimization, providing high-confidence performance guarantees.

Contribution

It proposes a novel approach that integrates parametric MDPs with scenario optimization to produce robust policies with quantifiable risk and performance guarantees.

Findings

Produces tight performance bounds with high confidence

Effectively handles multiple layers of environmental uncertainty

Demonstrates the approach on various benchmark problems

Abstract

We present a data-driven approach for producing policies that are provably robust across unknown stochastic environments. Existing approaches can learn models of a single environment as an interval Markov decision processes (IMDP) and produce a robust policy with a probably approximately correct (PAC) guarantee on its performance. However these are unable to reason about the impact of environmental parameters underlying the uncertainty. We propose a framework based on parametric Markov decision processes (MDPs) with unknown distributions over parameters. We learn and analyse IMDPs for a set of unknown sample environments induced by parameters. The key challenge is then to produce meaningful performance guarantees that combine the two layers of uncertainty: (1) multiple environments induced by parameters with an unknown distribution; (2) unknown induced environments which are approximated by IMDPs. We present a novel approach based on scenario optimisation that yields a single PAC guarantee quantifying the risk level for which a specified performance level can be assured in unseen environments, plus a means to trade-off risk and performance. We implement and evaluate our framework using multiple robust policy generation methods on a range of benchmarks. We show that our approach produces tight bounds on a policy's performance with high confidence.