BrainBench: Exposing the Commonsense Reasoning Gap in Large Language Models

TL;DR

BrainBench is a new benchmark of 100 questions designed to expose commonsense reasoning failures in large language models, revealing significant gaps between model performance and human-like reasoning.

Contribution

The paper introduces BrainBench, a comprehensive diagnostic benchmark targeting specific commonsense reasoning failure modes in LLMs, with extensive evaluation of leading models.

Findings

Top model achieves 80.3% accuracy

Models show a 6-16% gap between accuracy and consistency

Performance drops slightly in Chinese language evaluations

Abstract

Large language models (LLMs) achieve impressive scores on standard benchmarks yet routinely fail questions that any human would answer correctly in seconds. We introduce BrainBench, a benchmark of 100 brainteaser questions spanning 20 carefully designed categories, each targeting a specific commonsense reasoning failure mode in LLMs. Categories range from implicit physical constraints ("Should I walk or drive my rental car to the return lot?") to semantic scope tricks and default assumption hijacks. We evaluate eight frontier models -- four from the Claude family and four from the GPT family -- using a zero-shot protocol with 10 independent runs per question. The best model, Claude Opus 4.6 with extended thinking, achieves only 80.3% accuracy; the worst, GPT-4o, scores 39.7%. Even top-performing models exhibit a 6-16 percentage-point gap between accuracy and consistency, revealing stochastic reasoning. Cross-lingual evaluation in Chinese shows most models degrade by 2-8 percentage points, confirming that these failures reflect reasoning deficits rather than language-specific artifacts. BrainBench provides a fine-grained diagnostic tool for identifying where and why LLMs substitute surface heuristics for genuine commonsense reasoning.