Reinforcement Learning Friendly Vision-Language Model for Minecraft

TL;DR

This paper introduces CLIP4MC, a novel RL-friendly vision-language model for Minecraft that incorporates task completion into training, enabling better intrinsic rewards for open-ended tasks, with improved RL performance demonstrated on YouTube datasets.

Contribution

The paper proposes a new cross-modal contrastive learning framework, CLIP4MC, that integrates task completion signals into VLM training to enhance RL-friendliness for open-ended tasks.

Findings

Achieves better RL task performance than baselines.

Provides high-quality YouTube datasets for training.

Demonstrates effectiveness in Minecraft environment.

Abstract

One of the essential missions in the AI research community is to build an autonomous embodied agent that can achieve high-level performance across a wide spectrum of tasks. However, acquiring or manually designing rewards for all open-ended tasks is unrealistic. In this paper, we propose a novel cross-modal contrastive learning framework architecture, CLIP4MC, aiming to learn a reinforcement learning (RL) friendly vision-language model (VLM) that serves as an intrinsic reward function for open-ended tasks. Simply utilizing the similarity between the video snippet and the language prompt is not RL-friendly since standard VLMs may only capture the similarity at a coarse level. To achieve RL-friendliness, we incorporate the task completion degree into the VLM training objective, as this information can assist agents in distinguishing the importance between different states. Moreover, we provide neat YouTube datasets based on the large-scale YouTube database provided by MineDojo. Specifically, two rounds of filtering operations guarantee that the dataset covers enough essential information and that the video-text pair is highly correlated. Empirically, we demonstrate that the proposed method achieves better performance on RL tasks compared with baselines. The code and datasets are available at https://github.com/PKU-RL/CLIP4MC.