VLP: Vision-Language Preference Learning for Embodied Manipulation

TL;DR

This paper introduces VLP, a vision-language preference learning framework that uses implicit preference data to improve embodied manipulation tasks in reinforcement learning, reducing reliance on costly human annotations.

Contribution

The paper presents a novel vision-language preference model that learns from implicit data, enabling more efficient preference feedback in embodied manipulation tasks.

Findings

Outperforms baseline methods in simulated tasks

Generalizes well to unseen tasks and instructions

Provides accurate preference annotations

Abstract

Reward engineering is one of the key challenges in Reinforcement Learning (RL). Preference-based RL effectively addresses this issue by learning from human feedback. However, it is both time-consuming and expensive to collect human preference labels. In this paper, we propose a novel \textbf{V}ision-\textbf{L}anguage \textbf{P}reference learning framework, named \textbf{VLP}, which learns a vision-language preference model to provide preference feedback for embodied manipulation tasks. To achieve this, we define three types of language-conditioned preferences and construct a vision-language preference dataset, which contains versatile implicit preference orders without human annotations. The preference model learns to extract language-related features, and then serves as a preference annotator in various downstream tasks. The policy can be learned according to the annotated preferences via reward learning or direct policy optimization. Extensive empirical results on simulated embodied manipulation tasks demonstrate that our method provides accurate preferences and generalizes to unseen tasks and unseen language instructions, outperforming the baselines by a large margin.