PuzzleWorld: A Benchmark for Multimodal, Open-Ended Reasoning in Puzzlehunts

TL;DR

PuzzleWorld is a new benchmark with 667 multimodal, open-ended puzzles designed to evaluate reasoning capabilities of models, revealing current limitations and guiding future improvements.

Contribution

It introduces PuzzleWorld, a comprehensive benchmark with detailed annotations, and demonstrates the challenges faced by state-of-the-art models in open-ended reasoning tasks.

Findings

Most models achieve only 1-4% final answer accuracy.

The best model solves 18% of puzzles and reaches 40% stepwise accuracy.

Fine-tuning on reasoning traces improves stepwise accuracy from 4% to 11%.

Abstract

Puzzlehunts are a genre of complex, multi-step puzzles lacking well-defined problem definitions. In contrast to conventional reasoning benchmarks consisting of tasks with clear instructions and constrained environments, puzzlehunts requires discovering the underlying problem structure from multimodal evidence and iterative reasoning, mirroring real-world domains such as scientific discovery, exploratory data analysis, or investigative problem-solving. Despite progress in foundation models, their performance on open-ended settings remains largely untested. We introduce PuzzleWorld, a comprehensive benchmark of 667 puzzlehunt-style problems designed to assess step-by-step, open-ended, and creative multimodal reasoning. Each puzzle is annotated with the final solution, detailed reasoning traces, and cognitive skill labels, enabling holistic benchmarking and fine-grained diagnostic analysis. Most state-of-the-art models achieve only 1-4% final answer accuracy. On PuzzleWorld, the best model solves only 18% of puzzles and reaches 40% stepwise accuracy, matching human puzzle novices but falling significantly behind puzzle enthusiasts. To demonstrate the value of our reasoning annotations, we show that fine-tuning a small model on reasoning traces boosts stepwise accuracy from 4% to 11%, which translates to improvements in downstream visual reasoning tasks. Our detailed error analysis reveals that current models exhibit myopic reasoning, are bottlenecked by the limitations of language-based inference, and lack sketching capabilities crucial for visual and spatial reasoning. We release PuzzleWorld at https://github.com/MIT-MI/PuzzleWorld to support future work on building more general, open-ended, and creative reasoning systems.