Task-Induced Representation Learning

TL;DR

This paper compares unsupervised and task-induced representation learning methods for reinforcement learning in visually complex environments, showing that task-induced methods significantly improve sample efficiency and focus on task-relevant features.

Contribution

It introduces and evaluates task-induced representation learning, demonstrating its superiority over unsupervised methods in complex visual scenes for RL and imitation learning.

Findings

Task-induced representations double learning efficiency compared to unsupervised methods.

Representation learning improves sample efficiency in complex environments.

Task information helps focus on relevant scene parts, ignoring distractors.

Abstract

In this work, we evaluate the effectiveness of representation learning approaches for decision making in visually complex environments. Representation learning is essential for effective reinforcement learning (RL) from high-dimensional inputs. Unsupervised representation learning approaches based on reconstruction, prediction or contrastive learning have shown substantial learning efficiency gains. Yet, they have mostly been evaluated in clean laboratory or simulated settings. In contrast, real environments are visually complex and contain substantial amounts of clutter and distractors. Unsupervised representations will learn to model such distractors, potentially impairing the agent's learning efficiency. In contrast, an alternative class of approaches, which we call task-induced representation learning, leverages task information such as rewards or demonstrations from prior tasks to focus on task-relevant parts of the scene and ignore distractors. We investigate the effectiveness of unsupervised and task-induced representation learning approaches on four visually complex environments, from Distracting DMControl to the CARLA driving simulator. For both, RL and imitation learning, we find that representation learning generally improves sample efficiency on unseen tasks even in visually complex scenes and that task-induced representations can double learning efficiency compared to unsupervised alternatives. Code is available at https://clvrai.com/tarp.