Offline Reinforcement Learning with Pseudometric Learning

TL;DR

This paper introduces a novel pseudometric learning approach for offline reinforcement learning, enabling policies to stay close to logged data support, improving learning stability and performance in limited coverage scenarios.

Contribution

It proposes an iterative pseudometric learning method, extends it to function approximation, and integrates it into an actor-critic algorithm with a new bonus for offline RL.

Findings

Effective in hand manipulation tasks

Improves policy stability in limited data scenarios

Outperforms baseline offline RL methods

Abstract

Offline Reinforcement Learning methods seek to learn a policy from logged transitions of an environment, without any interaction. In the presence of function approximation, and under the assumption of limited coverage of the state-action space of the environment, it is necessary to enforce the policy to visit state-action pairs close to the support of logged transitions. In this work, we propose an iterative procedure to learn a pseudometric (closely related to bisimulation metrics) from logged transitions, and use it to define this notion of closeness. We show its convergence and extend it to the function approximation setting. We then use this pseudometric to define a new lookup based bonus in an actor-critic algorithm: PLOFF. This bonus encourages the actor to stay close, in terms of the defined pseudometric, to the support of logged transitions. Finally, we evaluate the method on hand manipulation and locomotion tasks.