Equilibrium in Misspecified Markov Decision Processes

TL;DR

This paper analyzes Markov decision processes where the agent's model is misspecified, developing an equilibrium concept to understand long-term behavior despite model uncertainty and updating beliefs via Bayes' rule.

Contribution

It introduces an equilibrium framework for misspecified Markov decision processes and characterizes steady state behavior under asymptotic analysis, extending static results to dynamic settings.

Findings

Equilibrium coincides with Berk-Nash equilibrium in static cases

Provides conditions for steady state characterization

Discusses issues related to negative experimentation value

Abstract

We study Markov decision problems where the agent does not know the transition probability function mapping current states and actions to future states. The agent has a prior belief over a set of possible transition functions and updates beliefs using Bayes' rule. We allow her to be misspecified in the sense that the true transition probability function is not in the support of her prior. This problem is relevant in many economic settings but is usually not amenable to analysis by the researcher. We make the problem tractable by studying asymptotic behavior. We propose an equilibrium notion and provide conditions under which it characterizes steady state behavior. In the special case where the problem is static, equilibrium coincides with the single-agent version of Berk-Nash equilibrium (Esponda and Pouzo (2016)). We also discuss subtle issues that arise exclusively in dynamic settings due to the possibility of a negative value of experimentation.