# Of Cores: A Partial-Exploration Framework for Markov Decision Processes

## TL;DR

This paper presents a framework for approximate analysis of Markov decision processes by identifying a core subsystem through simulations, enabling efficient analysis with rigorous error bounds across different horizon properties.

## Contribution

It introduces a novel partial-exploration framework for MDPs that uses simulation-based core identification to improve analysis efficiency and accuracy.

## Key findings

- Efficient analysis algorithms for MDPs with bounded, unbounded, and infinite horizons.
- High-probability core identification reduces computational complexity.
- Provides rigorous error bounds for approximate analysis.

## Abstract

We introduce a framework for approximate analysis of Markov decision processes (MDP) with bounded-, unbounded-, and infinite-horizon properties. The main idea is to identify a "core" of an MDP, i.e., a subsystem where we provably remain with high probability, and to avoid computation on the less relevant rest of the state space. Although we identify the core using simulations and statistical techniques, it allows for rigorous error bounds in the analysis. Consequently, we obtain efficient analysis algorithms based on partial exploration for various settings, including the challenging case of strongly connected systems.