Policy Adaptation from Foundation Model Feedback

TL;DR

This paper introduces PAFF, a method that uses foundation model feedback to automatically relabel demonstrations, significantly enhancing policy generalization to unseen objects, tasks, environments, and sim-to-real transfer in robotics.

Contribution

The paper presents a novel approach that leverages foundation model feedback for automatic demonstration relabeling to improve policy adaptation in robotics.

Findings

PAFF significantly outperforms baselines in generalization tasks.

The method improves sim-to-real transfer performance.

Demonstrates robustness across diverse unseen scenarios.

Abstract

Recent progress on vision-language foundation models have brought significant advancement to building general-purpose robots. By using the pre-trained models to encode the scene and instructions as inputs for decision making, the instruction-conditioned policy can generalize across different objects and tasks. While this is encouraging, the policy still fails in most cases given an unseen task or environment. In this work, we propose Policy Adaptation from Foundation model Feedback (PAFF). When deploying the trained policy to a new task or a new environment, we first let the policy play with randomly generated instructions to record the demonstrations. While the execution could be wrong, we can use the pre-trained foundation models to provide feedback to relabel the demonstrations. This automatically provides new pairs of demonstration-instruction data for policy fine-tuning. We evaluate our method on a broad range of experiments with the focus on generalization on unseen objects, unseen tasks, unseen environments, and sim-to-real transfer. We show PAFF improves baselines by a large margin in all cases. Our project page is available at https://geyuying.github.io/PAFF/