Make the Pertinent Salient: Task-Relevant Reconstruction for Visual Control with Distractions

TL;DR

This paper introduces Segmentation Dreamer (SD), an auxiliary task that improves visual control in distracting environments by reconstructing only task-relevant image components, enhancing sample efficiency and robustness.

Contribution

The paper proposes a novel segmentation-based auxiliary task for MBRL that isolates task-relevant features, improving perception and learning in visually distracting environments.

Findings

SD outperforms prior methods in distracted DMC and Meta-World tasks.

SD enables training in sparse reward scenarios previously unsolvable.

Using imperfect masks still yields significant performance gains.

Abstract

Recent advancements in Model-Based Reinforcement Learning (MBRL) have made it a powerful tool for visual control tasks. Despite improved data efficiency, it remains challenging to train MBRL agents with generalizable perception. Training in the presence of visual distractions is particularly difficult due to the high variation they introduce to representation learning. Building on DREAMER, a popular MBRL method, we propose a simple yet effective auxiliary task to facilitate representation learning in distracting environments. Under the assumption that task-relevant components of image observations are straightforward to identify with prior knowledge in a given task, we use a segmentation mask on image observations to only reconstruct task-relevant components. In doing so, we greatly reduce the complexity of representation learning by removing the need to encode task-irrelevant objects in the latent representation. Our method, Segmentation Dreamer (SD), can be used either with ground-truth masks easily accessible in simulation or by leveraging potentially imperfect segmentation foundation models. The latter is further improved by selectively applying the reconstruction loss to avoid providing misleading learning signals due to mask prediction errors. In modified DeepMind Control suite (DMC) and Meta-World tasks with added visual distractions, SD achieves significantly better sample efficiency and greater final performance than prior work. We find that SD is especially helpful in sparse reward tasks otherwise unsolvable by prior work, enabling the training of visually robust agents without the need for extensive reward engineering.