Automated Feedback Generation for Undergraduate Mathematics: Development and Evaluation of an AI Teaching Assistant

TL;DR

This paper introduces an AI-powered system that provides automated, high-quality feedback on free-form mathematical reasoning for undergraduate students, matching human expert assessments and adaptable for educational platforms.

Contribution

We develop a modular, editable AI system using large language models capable of evaluating and commenting on free-form mathematical proofs, including style and presentation, with deployment in a real educational platform.

Findings

Feedback quality comparable to human experts

System handles a wide range of edge cases in mathematical reasoning

Successful deployment on an educational platform

Abstract

Intelligent tutoring systems have long enabled automated immediate feedback on student work when it is presented in a tightly structured format and when problems are very constrained, but reliably assessing free-form mathematical reasoning remains challenging. We present a system that processes free-form natural language input, handles a wide range of edge cases, and comments competently not only on the technical correctness of submitted proofs, but also on style and presentation issues. We discuss the advantages and disadvantages of various approaches to the evaluation of such a system, and show that by the metrics we evaluate, the quality of the feedback generated is comparable to that produced by human experts when assessing early undergraduate homework. We stress-test our system with a small set of more advanced and unusual questions, and report both significant gaps and encouraging successes in that more challenging setting. Our system uses large language models in a modular workflow. The workflow configuration is human-readable and editable without programming knowledge, and allows some intermediate steps to be precomputed or injected by the instructor. A version of our tool is deployed on the Imperial mathematics homework platform Lambdafeedback. We report also on the integration of our tool into this platform.