Trajectory Modeling via Random Utility Inverse Reinforcement Learning

TL;DR

This paper introduces a novel inverse reinforcement learning framework for modeling driver trajectories using random utility theory, accounting for unobserved features and providing theoretical guarantees and a real-world case study.

Contribution

It proposes a deterministic policy-based inverse reinforcement learning model incorporating unobserved features via extended states, extending maximum entropy IRL and applying Bayesian inference.

Findings

Theoretical guarantees for the existence of solutions.

Maximum entropy IRL as a special case of the proposed model.

Successful application to real city trajectory data.

Abstract

We consider the problem of modeling trajectories of drivers in a road network from the perspective of inverse reinforcement learning. Cars are detected by sensors placed on sparsely distributed points on the street network of a city. As rational agents, drivers are trying to maximize some reward function unknown to an external observer. We apply the concept of random utility from econometrics to model the unknown reward function as a function of observed and unobserved features. In contrast to current inverse reinforcement learning approaches, we do not assume that agents act according to a stochastic policy; rather, we assume that agents act according to a deterministic optimal policy and show that randomness in data arises because the exact rewards are not fully observed by an external observer. We introduce the concept of extended state to cope with unobserved features and develop a Markov decision process formulation of drivers decisions. We present theoretical results which guarantee the existence of solutions and show that maximum entropy inverse reinforcement learning is a particular case of our approach. Finally, we illustrate Bayesian inference on model parameters through a case study with real trajectory data from a large city in Brazil.