The AlphaPhysics Term Rewriting System for Marking Algebraic Expressions in Physics Exams

TL;DR

This paper introduces an automated system combining language models, algebra, SMT solving, and term rewriting to evaluate physics exam answers, demonstrating effectiveness on real student responses.

Contribution

It presents a novel integrated approach using term rewriting and automated reasoning for physics answer grading, including the development of a specialized rewrite system with proven properties.

Findings

Successfully graded over 1500 real student responses

Demonstrated the effectiveness of the rewrite system in correctness assessment

Combined language models with formal methods for improved accuracy

Abstract

We present our method for automatically marking Physics exams. The marking problem consists in assessing typed student answers for correctness with respect to a ground truth solution. This is a challenging problem that we seek to tackle using a combination of a computer algebra system, an SMT solver and a term rewriting system. A Large Language Model is used to interpret and remove errors from student responses and rewrite these in a machine readable format. Once formalized and language-aligned, the next step then consists in applying automated reasoning techniques for assessing student solution correctness. We consider two methods of automated theorem proving: off-the-shelf SMT solving and term rewriting systems tailored for physics problems involving trigonometric expressions. The development of the term rewrite system and establishing termination and confluence properties was not trivial, and we describe it in some detail in the paper. We evaluate our system on a rich pool of over 1500 real-world student exam responses from the 2023 Australian Physics Olympiad.