Efficient Inference for Inverse Reinforcement Learning and Dynamic Discrete Choice Models

TL;DR

This paper introduces a semiparametric framework for efficient, debiased inference in inverse reinforcement learning and dynamic discrete choice models, enabling flexible nonparametric estimation with statistical guarantees.

Contribution

It develops a unified, computationally tractable approach for statistically efficient inference in IRL and DDC models using modern machine learning techniques.

Findings

Achieves $\,\sqrt{n}$-consistency and asymptotic normality.

Extends classical DDC inference to nonparametric rewards.

Provides automatic debiased estimators for policy value functionals.

Abstract

Inverse reinforcement learning (IRL) and dynamic discrete choice (DDC) models explain sequential decision-making by recovering reward functions that rationalize observed behavior. Flexible IRL methods typically rely on machine learning but provide no guarantees for valid inference, while classical DDC approaches impose restrictive parametric specifications and often require repeated dynamic programming. We develop a semiparametric framework for debiased inverse reinforcement learning that yields statistically efficient inference for a broad class of reward-dependent functionals in maximum entropy IRL and Gumbel-shock DDC models. We show that the log-behavior policy acts as a pseudo-reward that point-identifies policy value differences and, under a simple normalization, the reward itself. We then formalize these targets, including policy values under known and counterfactual softmax policies and functionals of the normalized reward, as smooth functionals of the behavior policy and transition kernel, establish pathwise differentiability, and derive their efficient influence functions. Building on this characterization, we construct automatic debiased machine-learning estimators that allow flexible nonparametric estimation of nuisance components while achieving $\sqrt{n}$-consistency, asymptotic normality, and semiparametric efficiency. Our framework extends classical inference for DDC models to nonparametric rewards and modern machine-learning tools, providing a unified and computationally tractable approach to statistical inference in IRL.