Little Impact of ChatGPT Availability on High School Student Test Score Performance

TL;DR

This study investigates the real-world impact of ChatGPT on high school test scores by analyzing usage patterns and finds no significant effect, suggesting AI's influence on learning outcomes may be minimal or balanced.

Contribution

It uniquely uses seasonal variations in ChatGPT activity to assess its actual educational impact outside controlled experiments.

Findings

No significant impact of AI usage on test scores in either direction.

AI use in education may be neutral overall, with positive and negative effects canceling out.

Real-world AI usage patterns differ from controlled study environments.

Abstract

In educational settings, AI can be used as a learning aid, but can also be used to avoid schoolwork, thereby passing classes while learning little. Many existing studies on the impact of AI on education focus on AI use in controlled settings or with specialized tools. In this paper, the dropoff in ChatGPT activity during non-school summer months in 2023 and 2024 is used to identify areas with heavy educational AI use and thus estimate the educational impact of AI as it is actually used. I find no meaningful impact of AI usage on high school test score averages in either direction. These results imply that, to the extent that high school students use AI to avoid learning, it either does not matter much for their test performance or is cancelled out by positive uses of AI in the aggregate.