Enhancing Robotic Manipulation with AI Feedback from Multimodal Large Language Models

TL;DR

This paper introduces CriticGPT, a multimodal large language model that analyzes robot manipulation videos to provide automated preference feedback, improving policy learning without task-specific details.

Contribution

The study presents CriticGPT, a novel multimodal LLM that acts as a critic for robot manipulation, enabling preference-based guidance solely from image inputs.

Findings

CriticGPT accurately predicts preferences and generalizes to new tasks.

It outperforms existing pre-trained models in guiding policy learning.

Effective in real-world robot manipulation scenarios.

Abstract

Recently, there has been considerable attention towards leveraging large language models (LLMs) to enhance decision-making processes. However, aligning the natural language text instructions generated by LLMs with the vectorized operations required for execution presents a significant challenge, often necessitating task-specific details. To circumvent the need for such task-specific granularity, inspired by preference-based policy learning approaches, we investigate the utilization of multimodal LLMs to provide automated preference feedback solely from image inputs to guide decision-making. In this study, we train a multimodal LLM, termed CriticGPT, capable of understanding trajectory videos in robot manipulation tasks, serving as a critic to offer analysis and preference feedback. Subsequently, we validate the effectiveness of preference labels generated by CriticGPT from a reward modeling perspective. Experimental evaluation of the algorithm's preference accuracy demonstrates its effective generalization ability to new tasks. Furthermore, performance on Meta-World tasks reveals that CriticGPT's reward model efficiently guides policy learning, surpassing rewards based on state-of-the-art pre-trained representation models.