Training-free Generation of Temporally Consistent Rewards from VLMs

TL;DR

This paper introduces $ ext{T}^2$-VLM, a training-free framework that generates accurate, temporally consistent rewards from vision-language models for robotic manipulation, improving decision-making and failure recovery without fine-tuning.

Contribution

The paper presents a novel training-free method that tracks subgoal status changes in VLMs to produce reliable rewards, enhancing RL performance in robotic tasks.

Findings

Achieves state-of-the-art results in robot manipulation benchmarks.

Provides accurate rewards with reduced computational costs.

Enhances long-horizon decision-making and failure recovery.

Abstract

Recent advances in vision-language models (VLMs) have significantly improved performance in embodied tasks such as goal decomposition and visual comprehension. However, providing accurate rewards for robotic manipulation without fine-tuning VLMs remains challenging due to the absence of domain-specific robotic knowledge in pre-trained datasets and high computational costs that hinder real-time applicability. To address this, we propose $\mathrm{T}^2$-VLM, a novel training-free, temporally consistent framework that generates accurate rewards through tracking the status changes in VLM-derived subgoals. Specifically, our method first queries the VLM to establish spatially aware subgoals and an initial completion estimate before each round of interaction. We then employ a Bayesian tracking algorithm to update the goal completion status dynamically, using subgoal hidden states to generate structured rewards for reinforcement learning (RL) agents. This approach enhances long-horizon decision-making and improves failure recovery capabilities with RL. Extensive experiments indicate that $\mathrm{T}^2$-VLM achieves state-of-the-art performance in two robot manipulation benchmarks, demonstrating superior reward accuracy with reduced computation consumption. We believe our approach not only advances reward generation techniques but also contributes to the broader field of embodied AI. Project website: https://t2-vlm.github.io/.