Watson & Holmes: A Naturalistic Benchmark for Comparing Human and LLM Reasoning

TL;DR

This paper introduces Watson & Holmes, a naturalistic reasoning benchmark based on a detective game, to evaluate AI reasoning in realistic narrative contexts, showing significant performance improvements over time.

Contribution

It presents a novel, scalable benchmark using a detective game with automated grading, enabling comparison of human and AI reasoning in naturalistic scenarios.

Findings

AI performance improved from lower quartile to top 5% over nine months.

Performance decline observed on longer cases with 1900-4000 words.

Early-stage reasoning models excelled at inductive reasoning with limited evidence.

Abstract

Existing benchmarks for AI reasoning provide limited insight into how closely these capabilities resemble human reasoning in naturalistic contexts. We present an adaptation of the Watson & Holmes detective tabletop game as a new benchmark designed to evaluate reasoning performance using incrementally presented narrative evidence, open-ended questions and unconstrained language responses. An automated grading system was developed and validated against human assessors to enable scalable and replicable performance evaluation. Results show a clear improvement in AI model performance over time. Over nine months of 2025, model performance rose from the lower quartile of the human comparison group to approximately the top 5%. Around half of this improvement reflects steady advancement across successive model releases, while the remainder corresponds to a marked step change associated with reasoning-oriented model architectures. Systematic differences in the performance of AI models compared to humans, dependent on features of the specific detection puzzle, were mostly absent with the exception of a fall in performance for models when solving longer cases (case lengths being in the range of 1900-4000 words), and an advantage at inductive reasoning for reasoning models at early stages of case solving when evidence was scant.