Affordance-Guided Reinforcement Learning via Visual Prompting

TL;DR

This paper introduces KAGI, a method that uses vision-language models to shape dense rewards for reinforcement learning, significantly improving sample efficiency and robustness in robotic manipulation tasks guided by natural language descriptions.

Contribution

KAGI leverages zero-shot affordance reasoning from vision-language models to provide dense rewards, enhancing autonomous RL performance in real-world manipulation tasks.

Findings

KAGI improves sample efficiency in robotic learning tasks.

KAGI enables successful task completion with fewer demonstrations.

KAGI maintains robustness even with reduced pre-training data.

Abstract

Robots equipped with reinforcement learning (RL) have the potential to learn a wide range of skills solely from a reward signal. However, obtaining a robust and dense reward signal for general manipulation tasks remains a challenge. Existing learning-based approaches require significant data, such as human demonstrations of success and failure, to learn task-specific reward functions. Recently, there is also a growing adoption of large multi-modal foundation models for robotics that can perform visual reasoning in physical contexts and generate coarse robot motions for manipulation tasks. Motivated by this range of capability, in this work, we present Keypoint-based Affordance Guidance for Improvements (KAGI), a method leveraging rewards shaped by vision-language models (VLMs) for autonomous RL. State-of-the-art VLMs have demonstrated impressive zero-shot reasoning about affordances through keypoints, and we use these to define dense rewards that guide autonomous robotic learning. On diverse real-world manipulation tasks specified by natural language descriptions, KAGI improves the sample efficiency of autonomous RL and enables successful task completion in 30K online fine-tuning steps. Additionally, we demonstrate the robustness of KAGI to reductions in the number of in-domain demonstrations used for pre-training, reaching similar performance in 45K online fine-tuning steps. Project website: https://sites.google.com/view/affordance-guided-rl