TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics

TL;DR

TOPReward introduces a novel probabilistic temporal value function that leverages pretrained vision-language models to estimate robotic task progress, significantly improving zero-shot reward estimation across diverse real-world tasks.

Contribution

It presents TOPReward, a new method that extracts task progress directly from VLM token logits, enhancing generalization and performance in robotic reinforcement learning.

Findings

Achieves 0.947 mean VOC on 130+ real-world tasks

Outperforms state-of-the-art GVL baseline

Enables downstream applications like success detection

Abstract

While Vision-Language-Action (VLA) models have seen rapid progress in pretraining, their advancement in Reinforcement Learning (RL) remains hampered by low sample efficiency and sparse rewards in real-world settings. Developing generalizable process reward models is essential for providing the fine-grained feedback necessary to bridge this gap, yet existing temporal value functions often fail to generalize beyond their training domains. We introduce TOPReward, a novel, probabilistically grounded temporal value function that leverages the latent world knowledge of pretrained video Vision-Language Models (VLMs) to estimate robotic task progress. Unlike prior methods that prompt VLMs to directly output progress values, which are prone to numerical misrepresentation, TOPReward extracts task progress directly from the VLM's internal token logits. In zero-shot evaluations across 130+ distinct real-world tasks and multiple robot platforms (e.g., Franka, YAM, SO-100/101), TOPReward achieves 0.947 mean Value-Order Correlation (VOC) on Qwen3-VL, dramatically outperforming the state-of-the-art GVL baseline which achieves near-zero correlation on the same open-source model. We further demonstrate that TOPReward serves as a versatile tool for downstream applications, including success detection and reward-aligned behavior cloning.