A Geometric Perspective on Self-Supervised Policy Adaptation

TL;DR

This paper introduces a geometric framework for long-term self-supervised policy adaptation in reinforcement learning, addressing real-world distractions and improving generalization through manifold manipulation.

Contribution

It presents a novel geometric perspective on self-supervised RL adaptation, analyzing embedding space dynamics and proposing methods to enhance long-term policy robustness.

Findings

Embedding space processes can negatively impact performance and can be mitigated.

Manipulating the geometry of actor and critic manifolds improves generalization.

Theoretical insights into actor-based and actor-free agent adaptation.

Abstract

One of the most challenging aspects of real-world reinforcement learning (RL) is the multitude of unpredictable and ever-changing distractions that could divert an agent from what was tasked to do in its training environment. While an agent could learn from reward signals to ignore them, the complexity of the real-world can make rewards hard to acquire, or, at best, extremely sparse. A recent class of self-supervised methods have shown promise that reward-free adaptation under challenging distractions is possible. However, previous work focused on a short one-episode adaptation setting. In this paper, we consider a long-term adaptation setup that is more akin to the specifics of the real-world and propose a geometric perspective on self-supervised adaptation. We empirically describe the processes that take place in the embedding space during this adaptation process, reveal some of its undesirable effects on performance and show how they can be eliminated. Moreover, we theoretically study how actor-based and actor-free agents can further generalise to the target environment by manipulating the geometry of the manifolds described by the actor and critic functions.