Central Limit Theorems for Transition Probabilities of Controlled Markov Chains

TL;DR

This paper establishes central limit theorems for estimators of transition probabilities and value functions in controlled Markov chains, providing statistical tools for offline policy evaluation and hypothesis testing.

Contribution

It introduces CLTs for transition matrices and value functions in controlled Markov chains, with conditions on logging policies and applications to goodness-of-fit tests.

Findings

CLTs for transition probability estimators under specific conditions

Asymptotic normality of value and Q-functions for stationary policies

Development of goodness-of-fit tests for logged data

Abstract

We develop a central limit theorem (CLT) for a non-parametric estimator of the transition matrices in controlled Markov chains (CMCs) with finite state-action spaces. Our results establish precise conditions on the logging policy under which the estimator is asymptotically normal, and reveal settings in which no CLT can exist. We then build on it to derive CLTs for the value, Q-, and advantage functions of any stationary stochastic policy, including the optimal policy recovered from the estimated model. Goodness-of-fit tests are derived as a corollary, which enable to test whether the logged data is stochastic. These results provide new statistical tools for offline policy evaluation and optimal policy recovery, and enable hypothesis tests for transition probabilities.