Active Learning of Markov Decision Processes using Baum-Welch algorithm (Extended)

TL;DR

This paper adapts the Baum-Welch algorithm for active learning of Markov decision processes, significantly reducing observation requirements for accurate model construction in cyber-physical systems.

Contribution

It introduces an active learning sampling strategy for MDPs based on Baum-Welch, improving efficiency over existing methods.

Findings

Active learning reduces the number of observations needed.

The approach outperforms state-of-the-art tools in empirical tests.

Significant improvements in model accuracy with fewer data.

Abstract

Cyber-physical systems (CPSs) are naturally modelled as reactive systems with nondeterministic and probabilistic dynamics. Model-based verification techniques have proved effective in the deployment of safety-critical CPSs. Central for a successful application of such techniques is the construction of an accurate formal model for the system. Manual construction can be a resource-demanding and error-prone process, thus motivating the design of automata learning algorithms to synthesise a system model from observed system behaviours. This paper revisits and adapts the classic Baum-Welch algorithm for learning Markov decision processes and Markov chains. For the case of MDPs, which typically demand more observations, we present a model-based active learning sampling strategy that choses examples which are most informative w.r.t.\ the current model hypothesis. We empirically compare our approach with state-of-the-art tools and demonstrate that the proposed active learning procedure can significantly reduce the number of observations required to obtain accurate models.