MDP Abstractions from Data: Large-Scale Stochastic Networks

TL;DR

This paper introduces a data-driven, compositional approach to construct finite MDPs for large-scale stochastic networks with unknown models, using dissipativity properties and probabilistic guarantees.

Contribution

It presents a novel framework combining dissipativity, stochastic storage functions, and data-driven methods to approximate large stochastic networks with finite MDPs and formal guarantees.

Findings

Successfully applied to a 100-room temperature network

Provides probabilistic bounds on trajectory differences

Demonstrates scalability to large networks

Abstract

This work proposes a compositional data-driven technique for the construction of finite Markov decision processes (MDPs) for large-scale stochastic networks with unknown mathematical models. Our proposed framework leverages dissipativity properties of subsystems and their finite MDPs using a notion of stochastic storage functions (SStF). In our data-driven scheme, we first build an SStF between each unknown subsystem and its data-driven finite MDP with a certified probabilistic confidence. We then derive dissipativity-type compositional conditions to construct a stochastic bisimulation function (SBF) between an interconnected network and its finite MDP using data-driven SStF of subsystems. Accordingly, we formally quantify the probabilistic distance between trajectories of an unknown large-scale stochastic network and those of its finite MDP with a guaranteed confidence. We illustrate the efficacy of our data-driven results over a room temperature network composing 100 rooms with unknown models.