Data-Driven Yet Formal Policy Synthesis for Stochastic Nonlinear Dynamical Systems

TL;DR

This paper introduces a data-driven method for synthesizing control policies for stochastic nonlinear systems, using learned interval MDP abstractions with PAC guarantees to handle nonlinearities and incomplete knowledge.

Contribution

It presents a novel data-driven abstraction technique for nonlinear stochastic systems that provides PAC guarantees and does not require complete system knowledge.

Findings

Effective in synthesizing reliable control policies under uncertainty.

Robust against incomplete knowledge of system dynamics.

Numerical experiments demonstrate practical applicability.

Abstract

The automated synthesis of control policies for stochastic dynamical systems presents significant challenges. A standard approach is to construct a finite-state abstraction of the continuous system, typically represented as a Markov decision process (MDP). However, generating abstractions is challenging when (1) the system's dynamics are nonlinear, and/or (2) we do not have complete knowledge of the dynamics. In this work, we introduce a novel data-driven abstraction technique for nonlinear Lipschitz continuous dynamical systems with additive stochastic noise that addresses both of these issues. As a key step, we use samples of the dynamics to learn the enabled actions and transition probabilities of the abstraction. We represent abstractions as MDPs with intervals of transition probabilities, known as interval MDPs (IMDPs). These abstractions enable the synthesis of policies for the concrete nonlinear system, with probably approximately correct (PAC) guarantees on the probability of satisfying a specified control objective. Our numerical experiments illustrate the effectiveness and robustness of our approach in achieving reliable control under uncertainty.