Target-Bench: Can Video World Models Achieve Mapless Path Planning with Semantic Targets?

TL;DR

Target-Bench is a new benchmark designed to evaluate the semantic reasoning, spatial estimation, and planning capabilities of video world models in robotic scenarios, revealing current limitations and potential improvements.

Contribution

We introduce Target-Bench, the first comprehensive benchmark for assessing semantic reasoning and planning in video world models, including a new evaluation metric and analysis of model performance.

Findings

Current models achieve only 0.341 overall score, indicating significant gaps.

Fine-tuning on small real-world datasets improves planning performance.

The benchmark covers 450 scenarios across 47 semantic categories.

Abstract

While recent video world models can generate highly realistic videos, their ability to perform semantic reasoning and planning remains unclear and unquantified. We introduce Target-Bench, the first benchmark that enables comprehensive evaluation of video world models' semantic reasoning, spatial estimation, and planning capabilities. Target-Bench provides 450 robot-collected scenarios spanning 47 semantic categories, with SLAM-based trajectories serving as motion tendency references. Our benchmark reconstructs motion from generated videos with a metric scale recovery mechanism, enabling the evaluation of planning performance with five complementary metrics that focus on target-approaching capability and directional consistency. Our evaluation result shows that the best off-the-shelf model achieves only a 0.341 overall score, revealing a significant gap between realistic visual generation and semantic reasoning in current video world models. Furthermore, we demonstrate that fine-tuning process on a relatively small real-world robot dataset can significantly improve task-level planning performance.