Generalization with Lossy Affordances: Leveraging Broad Offline Data for Learning Visuomotor Tasks

TL;DR

This paper presents a framework that leverages broad offline data and lossy representations to enable generalization and efficient learning of unseen visuomotor tasks in robotics, without manual reward engineering.

Contribution

It introduces a goal-conditioned policy learning method using offline reinforcement learning combined with online fine-tuning guided by lossy subgoal representations.

Findings

Effective task decomposition via affordance-based subgoals.

Improved generalization to unseen tasks using broad datasets.

Successful visual input-based fine-tuning without manual rewards.

Abstract

The utilization of broad datasets has proven to be crucial for generalization for a wide range of fields. However, how to effectively make use of diverse multi-task data for novel downstream tasks still remains a grand challenge in robotics. To tackle this challenge, we introduce a framework that acquires goal-conditioned policies for unseen temporally extended tasks via offline reinforcement learning on broad data, in combination with online fine-tuning guided by subgoals in learned lossy representation space. When faced with a novel task goal, the framework uses an affordance model to plan a sequence of lossy representations as subgoals that decomposes the original task into easier problems. Learned from the broad data, the lossy representation emphasizes task-relevant information about states and goals while abstracting away redundant contexts that hinder generalization. It thus enables subgoal planning for unseen tasks, provides a compact input to the policy, and facilitates reward shaping during fine-tuning. We show that our framework can be pre-trained on large-scale datasets of robot experiences from prior work and efficiently fine-tuned for novel tasks, entirely from visual inputs without any manual reward engineering.