Dissecting Physics Reasoning in Small Language Models: A Multi-Dimensional Analysis from an Educational Perspective

TL;DR

This study evaluates the reasoning reliability of small language models in physics education, revealing significant reasoning errors despite correct answers and highlighting the need for better evaluation methods focused on reasoning processes.

Contribution

Introduces Physbench and P-REFS for detailed, stage-wise evaluation of physics reasoning in small language models, exposing their reasoning failures and robustness issues.

Findings

75-98% of correct answers contain reasoning errors

Weaker models struggle with interpretation; stronger models fail during execution

Contextual variations minimally affect top models but impact mid-tier models

Abstract

Small Language Models (SLMs) offer privacy and efficiency for educational deployment, yet their utility depends on reliable multistep reasoning. Existing benchmarks often prioritize final answer accuracy, obscuring 'right answer, wrong procedure' failures that can reinforce student misconceptions. This work investigates SLM physics reasoning reliability, stage wise failure modes, and robustness under paired contextual variants. We introduce Physbench, comprising of 3,162 high school and AP level physics questions derived from OpenStax in a structured reference solution format with Bloom's Taxonomy annotations, plus 2,700 paired culturally contextualized variants. Using P-REFS, a stage wise evaluation rubric, we assess 10 SLMs across 58,000 responses. Results reveal substantial reliability gap: among final answer correct solutions, 75 to 98% contain at least one reasoning error. Failure modes shift with model capability; weaker models fail primarily at interpretation or modeling while stronger models often fail during execution. Paired contextual variations have minimal impact on top models but degrade the performance of mid-tier models. These findings demonstrate that safe educational AI requires evaluation paradigms that prioritize reasoning fidelity over final-answer correctness.