GPT takes the SAT: Tracing changes in Test Difficulty and Math Performance of Students

TL;DR

This study uses AI-generated control groups to analyze SAT math difficulty and student performance trends from 2008 to 2023, revealing a significant decline in both test rigor and student proficiency over time.

Contribution

Introduces 'Transformed Control', a novel AI-based method for creating control groups, enabling detailed analysis of SAT difficulty and performance trends over time.

Findings

SAT math difficulty decreased by 71 points from 2008 to 2023.

Student math performance declined by 36 points over the same period.

Disparities in performance decline observed among demographic groups.

Abstract

Scholastic Aptitude Test (SAT) is crucial for college admissions but its effectiveness and relevance are increasingly questioned. This paper enhances Synthetic Control methods by introducing "Transformed Control", a novel method that employs Large Language Models (LLMs) powered by Artificial Intelligence to generate control groups. We utilize OpenAI's API to generate a control group where GPT-4, or ChatGPT, takes multiple SATs annually from 2008 to 2023. This control group helps analyze shifts in SAT math difficulty over time, starting from the baseline year of 2008. Using parallel trends, we calculate the Average Difference in Scores (ADS) to assess changes in high school students' math performance. Our results indicate a significant decrease in the difficulty of the SAT math section over time, alongside a decline in students' math performance. The analysis shows a 71-point drop in the rigor of SAT math from 2008 to 2023, with student performance decreasing by 36 points, resulting in a 107-point total divergence in average student math performance. We investigate possible mechanisms for this decline in math proficiency, such as changing university selection criteria, increased screen time, grade inflation, and worsening adolescent mental health. Disparities among demographic groups show a 104-point drop for White students, 84 points for Black students, and 53 points for Asian students. Male students saw a 117-point reduction, while female students had a 100-point decrease.