R3L: Relative Representations for Reinforcement Learning

TL;DR

This paper introduces R3L, a framework that uses relative representations to enable reinforcement learning agents to adapt to new visual and task variations without retraining, reducing resource use.

Contribution

It adapts the relative representations framework to visual reinforcement learning, allowing for zero-shot generalization across different visual and task domains.

Findings

Enables agents to handle unseen visual-task pairs effectively

Reduces retraining time and computational resources

Demonstrates improved generalization in visual RL environments

Abstract

Visual Reinforcement Learning is a popular and powerful framework that takes full advantage of the Deep Learning breakthrough. It is known that variations in input domains (e.g., different panorama colors due to seasonal changes) or task domains (e.g., altering the target speed of a car) can disrupt agent performance, necessitating new training for each variation. Recent advancements in the field of representation learning have demonstrated the possibility of combining components from different neural networks to create new models in a zero-shot fashion. In this paper, we build upon relative representations, a framework that maps encoder embeddings to a universal space. We adapt this framework to the Visual Reinforcement Learning setting, allowing to combine agents components to create new agents capable of effectively handling novel visual-task pairs not encountered during training. Our findings highlight the potential for model reuse, significantly reducing the need for retraining and, consequently, the time and computational resources required.