Good-for-MDPs Automata for Probabilistic Analysis and Reinforcement Learning

TL;DR

This paper introduces 'good-for-MDPs' automata, a class suitable for probabilistic analysis and reinforcement learning, enabling effective state-space reduction and potentially improving learning outcomes.

Contribution

It characterizes GFM automata, shows their closure properties, and introduces a new class of low-branching Büchi automata that enhance reinforcement learning.

Findings

GFM automata are closed under simulation relations.

A new low-branching Büchi automata class is proposed.

Potential benefits for reinforcement learning beyond limit-deterministic automata.

Abstract

We characterize the class of nondeterministic ${\omega}$-automata that can be used for the analysis of finite Markov decision processes (MDPs). We call these automata `good-for-MDPs' (GFM). We show that GFM automata are closed under classic simulation as well as under more powerful simulation relations that leverage properties of optimal control strategies for MDPs. This closure enables us to exploit state-space reduction techniques, such as those based on direct and delayed simulation, that guarantee simulation equivalence. We demonstrate the promise of GFM automata by defining a new class of automata with favorable properties - they are B\"uchi automata with low branching degree obtained through a simple construction - and show that going beyond limit-deterministic automata may significantly benefit reinforcement learning.