Bayesian Inverse Transition Learning for Offline Settings

TL;DR

This paper introduces a Bayesian inverse transition learning method for offline reinforcement learning, focusing on reliable transition estimation, safety, and uncertainty quantification to improve policy performance and safety.

Contribution

It proposes a new constraint-based Bayesian approach to estimate transition dynamics, reducing variance and enhancing safety and informativeness of policies in offline RL.

Findings

High-performing policies with reduced variance

Effective uncertainty estimation for safer actions

Partial action ranking for improved planning

Abstract

Offline Reinforcement learning is commonly used for sequential decision-making in domains such as healthcare and education, where the rewards are known and the transition dynamics $T$ must be estimated on the basis of batch data. A key challenge for all tasks is how to learn a reliable estimate of the transition dynamics $T$ that produce near-optimal policies that are safe enough so that they never take actions that are far away from the best action with respect to their value functions and informative enough so that they communicate the uncertainties they have. Using data from an expert, we propose a new constraint-based approach that captures our desiderata for reliably learning a posterior distribution of the transition dynamics $T$ that is free from gradients. Our results demonstrate that by using our constraints, we learn a high-performing policy, while considerably reducing the policy's variance over different datasets. We also explain how combining uncertainty estimation with these constraints can help us infer a partial ranking of actions that produce higher returns, and helps us infer safer and more informative policies for planning.