From Canonical to Complex: Benchmarking LLM Capabilities in Undergraduate Thermodynamics

TL;DR

This paper evaluates the ability of large language models to understand and reason about undergraduate thermodynamics concepts, revealing current limitations in their competence for unsupervised educational use.

Contribution

The authors introduce UTQA, a comprehensive thermodynamics question benchmark, and analyze LLM performance, highlighting gaps in reasoning and visual understanding.

Findings

No model exceeded 95% accuracy threshold.

Best models achieved 82% accuracy, mostly on text-only questions.

Performance drops significantly on visual reasoning and irreversible processes.

Abstract

Large language models (LLMs) are increasingly considered as tutoring aids in science education. Yet their readiness for unsupervised use in undergraduate instruction remains uncertain, as reliable teaching requires more than fluent recall: it demands consistent, principle-grounded reasoning. Thermodynamics, with its compact laws and subtle distinctions between state and path functions, reversibility, and entropy, provides an ideal testbed for evaluating such capabilities. Here we present UTQA, a 50-item undergraduate thermodynamics question answering benchmark, covering ideal-gas processes, reversibility, and diagram interpretation. No leading 2025-era model exceeded our 95\% competence threshold: the best LLMs achieved 82\% accuracy, with text-only items performing better than image reasoning tasks, which often fell to chance levels. Prompt phrasing and syntactic complexity showed modest to little correlation with performance. The gap concentrates in finite-rate/irreversible scenarios and in binding visual features to thermodynamic meaning, indicating that current LLMs are not yet suitable for unsupervised tutoring in this domain.

Tables

Table 1. Table 1: Prompt categories and seventeen tested prompts with associated acronyms.

Category	Prompts (Acronyms)
Minimal Directive	BP, SP
Suppressed Reasoning	IRA, NoT
Reasoning Prompts	CoT, AdCoSM, Logical CoT, CoS, ToT, GoT, ThouT
Elimination Prompts	CoT-E, EliM
Persona / Expertise Amplifier	SIM
Affective / Framing Prompts	EI, EII, DIS

Table 2. Table 2: Illustration of degradation modes for Problem 8. Variants target three factors (clarity/accuracy, spelling/punctuation, and technical terminology) at two severities ( degraded = moderate, deficient = severe).

Version	Wording of question
Reference	Which process of an ideal gas is characterized by the following changes: $\Delta U=0$, $\Delta H=0$, and for system and environment, ?
Degraded (clarity & accuracy)	What type of gas process shows: $\Delta U=0$, $\Delta H=0$, and total for the system and surroundings?
Degraded (spelling & punctuation)	Which process of a ideal gas is characterised by the following changes $\Delta U=0$, $\Delta H=0$, and for system and environment
Degraded (technical terminology)	Which phase of gas behavior is marked by: $\Delta U=0$, $\Delta H=0$, and energy difference ( for everything involved?
Deficient (clarity & accuracy)	Which kind of gas process does it when: $\Delta U=0$, $\Delta H=0$, and total $\Delta G$ is going down for the gas and the rest?
Deficient (spelling & punctuation)	Wich proces of ideal gas is charactrised by these changes: $\Delta U=0$, $\Delta H=0$, and for system and enviroment,
Deficient (technical terminology)	What kind of gas step shows: $\Delta U=0$, $\Delta H=0$, and the energy change overall is negative for everything?

Full text

{tocentry}

uniwue]Institute of Physical and Theoretical Chemistry, Julius-Maximilian University Würzburg, 97074 Würzburg, Germany

uniwue]Institute of Physical and Theoretical Chemistry, Julius-Maximilian University Würzburg, 97074 Würzburg, Germany

uniwue]Institute of Physical and Theoretical Chemistry, Julius-Maximilian University Würzburg, 97074 Würzburg, Germany

uniwue]Institute of Physical and Theoretical Chemistry, Julius-Maximilian University Würzburg, 97074 Würzburg, Germany \altaffiliationBenchmark downloadable at: huggingface.co/datasets/herteltm/UTQA

1 Introduction

Large language models (LLMs) have emerged as highly capable general-purpose assistants, exhibiting impressive abilities to process, generate, and explain scientific content 1, 2, 3, 4, 5, 6. While much attention has focused on their potential for research support and knowledge retrieval, it remains unclear if their understanding is sufficient for unsupervised undergraduate-level tutoring. Reliable teaching requires more than fluent recall—it demands consistent reasoning that can guide students through complex problem-solving scenarios.

Here, thermodynamics provides an excellent opportunity for deeper evaluation. Its theoretical core is compact and well-established, yet correct application demands careful separation of crucial concepts: heat vs. work, state vs. process variables, and especially reversible vs. irreversible transformations—what Duhem called “the most important and, at the same time, most problematic to be defined in Thermodynamics” 7, 8, 9, 10, 11, 12. These distinctions are conceptually subtle yet mastering them requires both breadth and conceptual depth—spanning state functions and path dependencies.

Despite its foundational character, existing science benchmarks devote surprisingly little attention to thermodynamic reasoning. In GPQA 13, for instance, about 80% of chemistry questions concern organic recall, with almost no coverage of entropy or reversibility—a gap echoed in Humanity’s Last Exam 14. SciBench 15 is closer in spirit, assembling college-level physics, mathematics, and chemistry problems, yet its thermodynamics items are limited to quantitative end-answer calculations, with little exercise of reasoning about state functions, entropy bookkeeping, or reversibility. We note that gpt-5 correctly solved all 26 SciBench items we identified as topically aligned with our focus here. Given that entropy sets the arrow of time and limits energy conversion, and that reversibility underpins the second law (called unmatched among physical principles by Einstein 16), these omissions leave major gaps in evaluating LLM scientific reasoning.

We therefore introduce UTQA, a new 50-item single-choice benchmark specifically designed to evaluate LLM competence in undergraduate thermodynamics, with particular emphasis on ideal-gas processes, entropy, and reversibility. Rather than testing mere recall, our benchmark challenges models with multi-step reasoning problems including graphical representations that require integrating multiple constraints—the kind of problem-solving essential for effective tutoring.

Our initial results reveal both progress and persistent limitations. Current LLMs handle many straightforward text-only items reliably, yet performance drops sharply in two critical areas: finite-rate or irreversible scenarios that require nuanced thermodynamic reasoning, and diagram-based questions that demand mapping visual features to thermodynamic meaning. Notably, no model reached our provisional reliability threshold for unsupervised instructional use of 95% accuracy 17, highlighting a significant gap between fluent scientific explanation and the dependable, principle-grounded reasoning required for effective tutoring.

2 Methods

We conducted two classes of experiments. (i) Prompting and linguistic degradation experiments were run exclusively on gpt-4o via the API. (ii) The cross-model benchmark comparison was run on all other models using their command-line web interfaces under identical task settings. Full prompt texts and command invocations are provided in the Supporting Information (SI).

The benchmark comprises 50 single-choice items, each with three distractors and one correct answer option—33 text-only questions and 17 diagram-based questions. Prompting and linguistic degradation experiments as well as investigations of the role of linguistic question complexity were restricted to the text-only items to isolate verbal/semantic reasoning from visual interpretation. The diagram-based questions were analyzed separately in the cross-model comparison.

For each prompt variant, the text-only items were submitted in one or several independent runs in single-shot mode at sampling temperature $T=0.7$. To minimize context persistence and prompt-caching effects 18, we completed a full cycle of all 33 questions before repeating the set. This ordering suppressed spurious regularities (e.g., repetitive answer patterns) attributable to caching or internal batching rather than genuine model uncertainty.

We also evaluated 17 prompting strategies spanning minimal directives, suppressed-reasoning variants, structured reasoning (e.g., chain-of-thought), elimination-based prompts, persona framings, and affective/framing styles. Prompt wordings, acronyms, and attributions are listed in Table 1 and the SI. A two-phase variant (analysis followed by an explicit answer request) was also tested for comparison to single-shot prompting.

To quantify sensitivity to input quality, we applied controlled perturbations to a subset of items that were typically answered correctly under a simple baseline prompt. Three perturbation families were used: (i) reduced clarity (syntactic/grammatical distortions), (ii) orthographic noise (spelling/punctuation errors), and (iii) domain-terminology mismatches (systematic replacement with near-synonyms or colloquialisms). Each degraded item preserved the original physical content and correct answer key.

Accuracy per run is the fraction of correctly answered items. When multiple runs were performed, we report the mean accuracy $\bar{a}$ across runs. To separate genuine prompt effects from stochastic variability, we computed the run-to-run standard deviation $\sigma$ from residuals relative to each prompt’s mean and quote the uncertainty of the three-run mean as $\sigma_{3}=\sigma/\sqrt{3}$. Unless stated otherwise, comparisons between prompts are interpreted relative to $\sigma_{3}$.

All question texts, diagrams, answer keys, and problem solutions, are available in the project repository (see SI).

3 Benchmark design

Benchmark datasets are central to the development and evaluation of large language models (LLMs). Internal benchmarks, curated by model developers, guide training, track progress, and ensure alignment with specific goals. Popular external benchmarks (e.g., GPQA 13, HLE 14, HolisticEval 19, LLM-SRBench 20)—constructed independently to enable objective model comparison—have so far underrepresented thermodynamics and rarely probe core concepts such as entropy or reversibility (for a detailed breakdown of the GPQA diamond and HLE topical distributions, see SI). The SciBench benchmark 15 does include numerous thermodynamics items formulated as single-target calculations, but in our view it does not sufficiently probe thermodynamic reasoning, particularly around concepts such as reversibility. Our benchmark addresses these omissions directly.

3.1 Question Set Design

Our items were written to target clearly defined constructs in undergraduate thermodynamics while minimizing extraneous cognitive load. Each stem isolates a single concept or reasoning skill (e.g., state vs. path variables, sign conventions for $q$ and $w$, reversibility vs. finite-rate driving), and supplies only the context necessary for the intended inference. Diagrams such as the one shown in Fig. 1 are used only when they encode essential constraints and are rendered in an instructional, low-complexity style 21. Ambiguity is avoided by explicit units, sign conventions, and complete specifications of initial/final states or environmental contacts.

Content coverage spans the first and second laws, entropy changes of system and surroundings, pressure–volume work, heat transfer, and the distinction between quasistatic and non-quasistatic transformations. Items also probe recognition of path dependence, feasibility under the second law, and optimization principles in standard cycles (e.g., Carnot, Diesel). Several questions require translating between diagrammatic representations ($p$–$V$, $T$–$S$, $H$–$p$, $U$–$V$, $H$–$S$, $A$–$T$) and their thermodynamic meaning, including cases where axis labels are intentionally omitted to test structural understanding.

Both conceptual and quantitative formats are included, from process identification to multi-step constraints that combine energy and entropy balances. Numerical problems use parameter values chosen to prevent round-off ambiguities and to ensure a unique correct option. Distractors reflect common misconceptions (e.g., equating “adiabatic” with “reversible” or confusing $dU$ with $q$), so that accuracy reflects principled reasoning rather than elimination by superficial cues.

Item development followed an expert-driven, iterative process aligned with established guidance in educational measurement 22, 23. The English set was adapted from a previously reviewed German pool and refined across multiple rounds by subject-matter experts to enforce clarity, construct focus, and unambiguous phrasing. Model solutions were independently cross-checked for correctness and pedagogical soundness. Procedural details of modality split and experimental use are described in the Methods section.

3.2 Prompt Design and Prompting

LLM performance can depend strongly on prompt form and linguistic quality 24, 25, 26. We therefore varied prompt phrasing, structure, and tone to probe sensitivity on text-only items (isolating semantic reasoning from visual interpretation; see SI for protocol and uncertainty handling).

Seventeen prompts were tested (Table 1), spanning: minimal directives; suppressed-reasoning variants; structured reasoning prompts (CoT, ToT, GoT, Logical CoT, CoS) 27, 28, 29, 30, 31; elimination-style prompts; persona/expertise frames; and affective framings 32, 33. A two-phase “analyze then answer” format was included following 5. Full wordings and attributions are given in the SI. We included both explicit- and no-explanation variants to probe claims about unfaithful explanations and chain fragility 34, 35, and reasoning without explicit scaffolds 36.

To assess robustness to input quality, we additionally applied controlled linguistic degradations to items typically solved under a baseline prompt: (i) syntactic/grammatical distortions, (ii) orthographic noise (spelling/punctuation), and (iii) domain-terminology substitutions. These edits preserved physical content and the correct key (details in Methods).

Mean accuracies were computed over independent runs with run-to-run scatter summarized by $\sigma$ and the three-run mean uncertainty $\sigma_{3}$ (definitions in Methods). Results by prompt family and the impact of linguistic degradation are discussed in the Results section.

4 Results and Discussion

4.1 Prompting effects and uncertainty analysis

The 17 prompt variants were evaluated on the text-only items and quantified run-to-run scatter to separate genuine prompt effects from stochastic variability. The distribution of residuals across $17\times 3=51$ batches yields $\sigma=0.05$, corresponding to $\sigma_{3}\approx 0.03$ for three-run means (Fig. 2a). Mean accuracies per prompt span $0.36$–$0.54$ (Fig. 2b), a range that materially exceeds $\sigma_{3}$, indicating real prompt sensitivity.

Sorting by accuracy (Fig. 3a) shows two broad bands: higher-performing variants (e.g., SP, CoT, EI, ThouT, EII, SIM) and lower-performing ones (e.g., CoT-E, NoT, AdCoSM). Minimal prompts (BP/SP) perform comparably to several structured-reasoning formats, consistent with reports that models can recruit internal reasoning without explicit scaffolds 36. By contrast, elimination-style prompts underperform on this benchmark.

Several factors likely contribute to these differences. First, longer reasoning chains introduce more opportunities for error, and mistakes that occur late in a chain may be especially damaging, a phenomenon sometimes termed “late-stage fragility” 35. Second, explanations generated under chain-of-thought prompting may not faithfully reflect the model’s underlying computation 34; suppressing or elaborating such reasoning (NoT, IRA, AdCoSM) therefore has limited effect. Finally, persona or affective framings can alter tone and verbosity but rarely change substantive reasoning behavior 33. Together, these points suggest that prompt design primarily affects surface presentation and stability, but does little to correct deeper deficits in scientific reasoning.

A two-phase “analyze then answer” format 5 produced no statistically significant gains over single-shot prompting within $\sigma_{3}$, suggesting that explicit separation of analysis and answer does not improve accuracy for these items using gpt-4o.

4.2 Effect of Linguistic Degradation

We tested robustness to degraded wording under a fixed baseline prompt (Please answer the following single-choice question.), altering only the stem and options. Controlled edits targeted three dimensions as shown in Fig. 4— each at two severities (degraded, deficient). Edits preserved physical content and the correct answer key; the rubric used to generate variants (via gpt-4o) is provided in the SI. Table 2 illustrates all variants for a representative item.

Aggregate results over text-only items are shown in Fig. 4. For clarity/accuracy, accuracy declined from $0.52$ (reference) to $0.40$ (degraded) and $0.43$ (deficient). For spelling/punctuation, the degraded variant matched the reference ($0.52$), while the deficient variant dropped to $0.41$. For technical terminology, the degraded score was essentially unchanged ($0.53$) and declined for the deficient variant ($0.49$). Means are based on ten runs per condition with error bars indicating $\sigma_{10}\!\approx\!0.015$.

Two patterns are salient for educators. First, obscuring logical structure (clarity/accuracy) leads to the largest performance losses, even at moderate severity. Second, models tolerate moderate orthographic and terminology noise, but severe errors in either dimension degrade accuracy. The weak sensitivity to terminology substitutions suggests that, for domain-familiar material, models often recover meaning from context; by contrast, degraded clarity removes essential constraints and impairs reasoning. These findings align with established comprehension factors in educational measurement 22, 37.

4.3 Cross-model comparison

Using the same baseline prompt across platforms (Please answer the following single-choice question), we evaluated 19 contemporary models (APIs and web CLI) on both the text-only and diagram-based items.

For the text-only subset (Fig. 5, mean accuracies span $0.20$ (gpt-3.5-turbo-0125) to $0.88$ (DeepSeek R1), with a grand mean of $0.67$. With per-condition scatter $\sigma\!\approx\!0.05$, pairwise gaps of $\gtrsim 0.10$ are unlikely to reflect sampling variance alone. The distribution is stratified: legacy gpt-3.5 trails newer systems, while top-tier models (e.g., gpt-5 high-effort, Gemini 2.5 Pro, Grok 3 Think, gpt-o3, DeepSeek R1) exceed $0.80$. Within families, higher “reasoning budget” configurations outperform lighter variants (e.g., gpt-5 high vs. nano), consistent with broader evidence that capacity devoted to multi-step inference improves domain problem solving 38, 36.

4.4 Items involving diagram interpretation

Diagrams are central to thermodynamics instruction and problem solving: they externalize constraints, make path relations perceptually accessible, and support rapid inference 39, 40, 41. Performance dropped sharply when items required interpreting such diagrams (Fig. 6). Across 19 models, the mean accuracy was $32\%$—less than half the text-only mean ($67\%$). The weakest result (gpt-4.1) was $6\%$ suggesting consistently chosen detractors, while the strongest (gpt-o3) reached $76\%$; Gemini 2.5 Pro and gpt-o1 scored $54\%$ and $53\%$, respectively.

Two observations suggest that errors arise chiefly in binding visual features to thermodynamic meaning rather than in low-level recognition. First, separate probes indicated that models typically identify axes, reference markers, and basic curve characteristics (segmentation, linearity, concavity). Second, many mistakes reflect failures to integrate diagram structure with governing relations (e.g., area under $p$–$V$ paths as work; feasibility under the second law; path ordering across segments). In contrast, trained humans can often judge work or entropy trends by quick perceptual comparison of path geometry, a classic advantage of visual reasoning 39, 40.

4.5 Effect of linguistic complexity

Lastly we examined whether surface complexity of wording predicts accuracy on text-only items by using a simple proxy: the total number of clauses in each complete problem (stem $+$ options), where a clause contains a subject and a verb 42. For each question we computed mean accuracy across all tested models. As shown in Fig. 7 no significant correlation emerged over the observed range of $1$–$20$ clauses.

This negative result indicates that, within typical instructional phrasing, clause count is not the limiting factor for current LLMs; failures more plausibly reflect weaknesses in conceptual integration rather than syntactic load 43. Thus, for benchmark construction, varying clause count alone is unlikely to bias outcomes. Future analyses could evaluate richer textual metrics (e.g., referential cohesion, ambiguity) and their interaction with thermodynamic reasoning demands 42.

4.6 Common strengths and weaknesses

Across the two item classes (text-only and diagram), we observe a marked asymmetry. On text-only problems, the strongest recent models approach the level of a well-prepared graduate tutor, whereas performance on diagram-based items is broadly poor; the notable exception being gpt-o3, which consistently handles diagram-centric tasks better than other models.

High-scoring text items share a canonical structure: they reduce to a single state-function argument or a standard identity directly applicable without multi-constraint coupling. Typical successes include recognizing that for ideal gases $U$ and $H$ depend only on $T$, that throttling preserves $H$, and that entropy increases in free expansion. Distractors encoding familiar misconceptions are routinely eliminated, indicating robust recall of first-principles relations and sign conventions.

However, accuracy degrades when solutions require integrating multiple constraints, enforcing feasibility bounds, or reasoning about finite-rate (non-quasistatic) processes. For example, in a millisecond compression of a monatomic ideal gas to $V/2$, the bath is effectively adiabatic, and irreversible phenomena such as shock-front dissipation imply that $T_{1}/T_{0}$ must exceed the reversible-adiabatic value $2^{2/3}$; the strongest models increasingly identify the reversible result as a lower bound, correctly attributing the excess temperature to dissipation.

The most consistent error patterns are: (i) misuse of quasistatic templates despite explicit finite-rate cues; (ii) entropy bookkeeping errors—confusing transferred entropy with entropy production and applying $\Delta S$ formulas outside their domain; (iii) path-dependence blind spots for work—failing to reason with oriented areas in the $p\!-\!V$ plane and mixing sign conventions; (iv) missed invariants and feasibility constraints in optimization; and (v) numeric anchoring to textbook exponents/constants without checking applicability conditions.

On diagram-based items, many models can, when prompted, parse low-level features (axis labels, start/end states, segmentation, curvature/monotonicity). Errors arise at the binding stage: mapping percepts to thermodynamic quantities and constraints. Recurrent issues include computing and comparing signed areas $\int p\,\mathrm{d}V$ with correct orientation, binding leg types to axes (e.g., isochoric $\leftrightarrow$ vertical in $p\!-\!V$), enforcing feasibility across concatenated legs, and propagating state limits through a cycle. Template recognition (e.g., Otto/Diesel/Stirling) or simple sign-based eliminations yield acceptable performance, but tasks demanding compositional reasoning—integrating geometry (areas, slopes, orientations) with laws and state relations—remain the principal bottleneck.

5 Conclusions

We introduced UTQA, a small and deceptively simple benchmark in undergraduate thermodynamics: fifty single-choice items on ideal-gas processes and thermodynamic diagrams (two thirds text-only, one third diagram-based). Applied to current LLMs, the benchmark highlights both solid performance on many canonical items and persistent weaknesses in more demanding cases. No 2025-era model reached our 95% reliability threshold for unsupervised tutoring; even the top performer (gpt-o3 at 82%) fell well short of this target (Fig. 8).

The performance shortfall is not uniform but concentrates in two domains. (i) Finite-rate, irreversible scenarios expose fragile regime recognition: when dissipation, feasibility bounds, or non-quasistatic driving matter, accuracies drop significantly. (ii) Diagram-based items reveal a fundamental multimodal binding deficit: models can parse axes and curve features but consistently fail to map geometry to thermodynamic meaning (e.g., signed $\int p\,\mathrm{d}V$ as work, process classification, constraint consistency across cycles).

These deficits are particularly striking because graphical reasoning is where humans gain considerable efficiency. However, despite current limitations, this study documents substantial progress—the strongest 2025-era systems demonstrate solid macroscopic thermodynamics with increasingly consistent microscopic narratives, representing what we would reasonably characterize as deep understanding for many items. This suggests that—contingent on improved coupling of visual perception with physical constraints—reaching our accuracy threshold on this benchmark is plausible in the foreseeable future, a promising prospect for educational applications.

We note that meeting accuracy thresholds is necessary but insufficient for effective tutoring. Beyond reliable problem solving, systems must also deliver appropriate interaction granularity, timely feedback, and disambiguation in dialogue—capabilities not assessed here. Our scope is intentionally narrow (ideal gases; excluding real-gas effects, mixtures, phase equilibria, and transport), so additional failure modes may emerge under broader coverage that could further challenge current capabilities.

Future benchmark extensions toward real-gas behavior, mixtures, phase diagrams, and standard cycles will probe reasoning under richer thermodynamic constraints and help rebalance coverage relative to other well-benchmarked domains like quantum mechanics. More generally, discipline-specific benchmarks that encode conceptual bottlenecks provide valuable tools for measuring principled application rather than recall alone. As models improve on such carefully constructed items—and especially on multimodal binding and irreversible regimes—they move closer to becoming trustworthy, discipline-aware educational partners.

6 Acknowledgements

We thank Tobias Brixner, Ingo Fischer, and Roland Mitrić for their validation of a German-language predecessor to this benchmark.