Bayesian regularization of empirical MDPs

TL;DR

This paper introduces Bayesian regularization techniques for empirical MDPs to improve policy robustness against model noise, using $L^1$ and relative entropic regularization, validated on synthetic and real-world data.

Contribution

It proposes two novel Bayesian regularization methods for empirical MDPs, enhancing policy robustness and generalization over traditional approaches.

Findings

Regularized policies outperform unregularized ones in noisy environments

Both $L^1$ and entropic regularizations improve robustness

Empirical results on real-world shopping data confirm effectiveness

Abstract

In most applications of model-based Markov decision processes, the parameters for the unknown underlying model are often estimated from the empirical data. Due to noise, the policy learnedfrom the estimated model is often far from the optimal policy of the underlying model. When applied to the environment of the underlying model, the learned policy results in suboptimal performance, thus calling for solutions with better generalization performance. In this work we take a Bayesian perspective and regularize the objective function of the Markov decision process with prior information in order to obtain more robust policies. Two approaches are proposed, one based on $L^1$ regularization and the other on relative entropic regularization. We evaluate our proposed algorithms on synthetic simulations and on real-world search logs of a large scale online shopping store. Our results demonstrate the robustness of regularized MDP policies against the noise present in the models.