Adaptive Estimation of the Transition Density of Controlled Markov Chains

TL;DR

This paper introduces an adaptive, assumption-free method for estimating transition densities in controlled Markov chains, applicable in various fields like reinforcement learning and time series analysis.

Contribution

It develops a novel estimator that does not require prior smoothness knowledge or control distribution assumptions, advancing non-parametric density estimation in controlled Markovian settings.

Findings

Provides oracle risk bounds for the estimator

Validates the method with both randomized and deterministic loss functions

Offers a flexible framework for transition density estimation

Abstract

Estimating the transition dynamics of controlled Markov chains is crucial in fields such as time series analysis, reinforcement learning, and system exploration. Traditional non-parametric density estimation methods often assume independent samples and require oracle knowledge of smoothness parameters like the H\"older continuity coefficient. These assumptions are unrealistic in controlled Markovian settings, especially when the controls are non-Markovian, since such parameters need to hold uniformly over all control values. To address this gap, we propose an adaptive estimator for the transition densities of controlled Markov chains that does not rely on prior knowledge of smoothness parameters or assumptions about the control sequence distribution. Our method builds upon recent advances in adaptive density estimation by selecting an estimator that minimizes a loss function {and} fitting the observed data well, using a constrained minimax criterion over a dense class of estimators. We validate the performance of our estimator through oracle risk bounds, employing both randomized and deterministic versions of the Hellinger distance as loss functions. This approach provides a robust and flexible framework for estimating transition densities in controlled Markovian systems without imposing strong assumptions.