ABench-Physics: Benchmarking Physical Reasoning in LLMs via High-Difficulty and Dynamic Physics Problems

TL;DR

ABench-Physics introduces a challenging benchmark for evaluating large language models' physical reasoning and generalization abilities through static and dynamic physics problems of graduate and Olympiad difficulty.

Contribution

The paper presents ABench-Physics, a new benchmark with static and dynamic problems to rigorously assess LLMs' physics reasoning and robustness, addressing limitations of previous benchmarks.

Findings

State-of-the-art LLMs perform poorly on the benchmark.

Models struggle with dynamic problem variants and generalization.

Benchmark reveals significant gaps in physical reasoning capabilities.

Abstract

Large Language Models (LLMs) have shown impressive performance in domains such as mathematics and programming, yet their capabilities in physics remain underexplored and poorly understood. Physics poses unique challenges that demand not only precise computation but also deep conceptual understanding and physical modeling skills. Existing benchmarks often fall short due to limited difficulty, multiple-choice formats, and static evaluation settings that fail to capture physical modeling ability. In this paper, we introduce ABench-Physics, a novel benchmark designed to rigorously evaluate LLMs' physical reasoning and generalization capabilities. ABench-Physics consists of two components: Phy_A, a static set of 400 graduate- or Olympiad-level problems; and Phy_B, a dynamic subset of 100 problems equipped with an automatic variation engine to test model robustness across changing conditions. All questions require precise numerical answers, with strict formatting and tolerance constraints. Our evaluation of several state-of-the-art LLMs reveals substantial performance gaps, highlighting persistent limitations in physical reasoning, especially in generalization to dynamic variants. ABench-Physics provides a challenging and diagnostic framework for advancing scientific reasoning in LLMs.