The Impact of AI in Physics Education: A Comprehensive Review from GCSE to University Levels

TL;DR

This comprehensive review evaluates AI's capabilities in Physics education across GCSE to university levels, highlighting its strengths in early stages and vulnerabilities at advanced levels, with implications for assessment and policy.

Contribution

The study provides a detailed assessment of LLM performance in Physics education, offering new insights into its strengths, limitations, and policy recommendations for AI integration.

Findings

LLM scores highest on GCSE questions, with 83.4% accuracy.

Performance declines at higher educational levels, with 37.4% accuracy at university.

LLM shows potential in writing Physics essays and coding, but struggles with complex calculations.

Abstract

With the rapid evolution of Artificial Intelligence (AI), its potential implications for higher education have become a focal point of interest. This study delves into the capabilities of AI in Physics Education and offers actionable AI policy recommendations. Using a Large Language Model (LLM), we assessed its ability to answer 1337 Physics exam questions spanning GCSE, A-Level, and Introductory University curricula. We employed various AI prompting techniques: Zero Shot, In Context Learning, and Confirmatory Checking, which merges Chain of Thought reasoning with Reflection. The AI's proficiency varied across academic levels: it scored an average of 83.4% on GCSE, 63.8% on A-Level, and 37.4% on university-level questions, with an overall average of 59.9% using the most effective prompting technique. In a separate test, the LLM's accuracy on 5000 mathematical operations was found to decrease as the number of digits increased. Furthermore, when evaluated as a marking tool, the LLM's concordance with human markers averaged at 50.8%, with notable inaccuracies in marking straightforward questions, like multiple-choice. Given these results, our recommendations underscore caution: while current LLMs can consistently perform well on Physics questions at earlier educational stages, their efficacy diminishes with advanced content and complex calculations. LLM outputs often showcase novel methods not in the syllabus, excessive verbosity, and miscalculations in basic arithmetic. This suggests that at university, there's no substantial threat from LLMs for non-invigilated Physics questions. However, given the LLMs' considerable proficiency in writing Physics essays and coding abilities, non-invigilated examinations of these skills in Physics are highly vulnerable to automated completion by LLMs. This vulnerability also extends to Physics questions pitched at lower academic levels.