LLM-WikiRace Benchmark: How Far Can LLMs Plan over Real-World Knowledge Graphs?

TL;DR

LLM-Wikirace is a benchmark testing large language models' ability to plan and reason over real-world knowledge graphs, revealing significant challenges in complex, hard-level tasks despite strong performance on easier levels.

Contribution

This paper introduces LLM-Wikirace, a novel benchmark for evaluating planning and reasoning in LLMs using Wikipedia navigation tasks, highlighting current limitations and the importance of world knowledge.

Findings

Models perform well on easy tasks but struggle on hard ones.

Performance drops sharply on complex, long-horizon planning tasks.

Even top models often fail to replan after errors, entering loops.

Abstract

We introduce LLM-Wikirace, a benchmark for evaluating planning, reasoning, and world knowledge in large language models (LLMs). In LLM-Wikirace, models must efficiently navigate Wikipedia hyperlinks step by step to reach a target page from a given source, requiring look-ahead planning and the ability to reason about how concepts are connected in the real world. We evaluate a broad set of open- and closed-source models, including Gemini-3, GPT-5, and Claude Opus 4.5, which achieve the strongest results on the easy level of the task and demonstrate superhuman performance. Despite this, performance drops sharply on hard difficulty: the best-performing model, Gemini-3, succeeds in only 23\% of hard games, highlighting substantial remaining challenges for frontier models. Our analysis shows that world knowledge is a necessary ingredient for success, but only up to a point, beyond this threshold, planning and long-horizon reasoning capabilities become the dominant factors. Trajectory-level analysis further reveals that even the strongest models struggle to replan after failure, frequently entering loops rather than recovering. LLM-Wikirace is a simple benchmark that reveals clear limitations in current reasoning systems, offering an open arena where planning-capable LLMs still have much to prove. Our code and leaderboard available at https:/llmwikirace.github.io.