Bayesian Inverse Transition Learning: Learning Dynamics From Near-Optimal Trajectories

TL;DR

This paper introduces a Bayesian inverse transition learning method that estimates transition dynamics from near-optimal trajectories, improving decision-making in reinforcement learning and healthcare scenarios.

Contribution

It presents a novel constraint-based Bayesian approach that leverages expert trajectory coverage and near-optimality to accurately infer transition dynamics.

Findings

Significant improvements in decision-making in synthetic and healthcare environments.

Posterior estimates can predict transfer success in reinforcement learning.

Method outperforms existing approaches in estimating transition models.

Abstract

We consider the problem of estimating the transition dynamics $T^*$ from near-optimal expert trajectories in the context of offline model-based reinforcement learning. We develop a novel constraint-based method, Inverse Transition Learning, that treats the limited coverage of the expert trajectories as a \emph{feature}: we use the fact that the expert is near-optimal to inform our estimate of $T^*$. We integrate our constraints into a Bayesian approach. Across both synthetic environments and real healthcare scenarios like Intensive Care Unit (ICU) patient management in hypotension, we demonstrate not only significant improvements in decision-making, but that our posterior can inform when transfer will be successful.