Efficient Embedding of Semantic Similarity in Control Policies via Entangled Bisimulation

TL;DR

This paper introduces entangled bisimulation, a novel scalable metric for learning invariant representations in reinforcement learning, improving policy generalization amidst visual distractions.

Contribution

It proposes entangled bisimulation, a new unbiased, scalable bisimulation metric for continuous spaces, enhancing policy robustness against distractions.

Findings

Outperforms previous methods on Distracting Control Suite

Improves policy generalization with data augmentation

Provides a scalable, unbiased estimation of bisimulation metrics

Abstract

Learning generalizeable policies from visual input in the presence of visual distractions is a challenging problem in reinforcement learning. Recently, there has been renewed interest in bisimulation metrics as a tool to address this issue; these metrics can be used to learn representations that are, in principle, invariant to irrelevant distractions by measuring behavioural similarity between states. An accurate, unbiased, and scalable estimation of these metrics has proved elusive in continuous state and action scenarios. We propose entangled bisimulation, a bisimulation metric that allows the specification of the distance function between states, and can be estimated without bias in continuous state and action spaces. We show how entangled bisimulation can meaningfully improve over previous methods on the Distracting Control Suite (DCS), even when added on top of data augmentation techniques.