What Can I Do Here? Learning New Skills by Imagining Visual Affordances

TL;DR

This paper introduces visuomotor affordance learning (VAL), a method enabling robots to rapidly adapt to new environments by sampling and training on potential outcomes using generative models, improving zero-shot generalization.

Contribution

The paper presents a novel approach using generative models to learn visual affordances, allowing robots to sample possible outcomes and quickly adapt to new tasks with minimal online training.

Findings

VAL enables rapid adaptation to new environments.

Robots can learn new manipulation skills with only five minutes of online data.

The approach improves zero-shot generalization in real-world tasks.

Abstract

A generalist robot equipped with learned skills must be able to perform many tasks in many different environments. However, zero-shot generalization to new settings is not always possible. When the robot encounters a new environment or object, it may need to finetune some of its previously learned skills to accommodate this change. But crucially, previously learned behaviors and models should still be suitable to accelerate this relearning. In this paper, we aim to study how generative models of possible outcomes can allow a robot to learn visual representations of affordances, so that the robot can sample potentially possible outcomes in new situations, and then further train its policy to achieve those outcomes. In effect, prior data is used to learn what kinds of outcomes may be possible, such that when the robot encounters an unfamiliar setting, it can sample potential outcomes from its model, attempt to reach them, and thereby update both its skills and its outcome model. This approach, visuomotor affordance learning (VAL), can be used to train goal-conditioned policies that operate on raw image inputs, and can rapidly learn to manipulate new objects via our proposed affordance-directed exploration scheme. We show that VAL can utilize prior data to solve real-world tasks such drawer opening, grasping, and placing objects in new scenes with only five minutes of online experience in the new scene.