Whose ChatGPT? Unveiling Real-World Educational Inequalities Introduced by Large Language Models

TL;DR

This study analyzes real-world data to assess how large language models like ChatGPT impact educational inequalities, revealing nuanced effects that benefit some students more than others and emphasizing the importance of responsible implementation.

Contribution

It provides the first large-scale empirical analysis of LLM effects on educational equity using actual student data from a diverse university setting.

Findings

Overall writing quality improved post-LLM availability

Gaps between advantaged and disadvantaged students narrowed

Higher socioeconomic students benefited more from LLMs

Abstract

The universal availability of ChatGPT and other similar tools since late 2022 has prompted tremendous public excitement and experimental effort about the potential of large language models (LLMs) to improve learning experience and outcomes, especially for learners from disadvantaged backgrounds. However, little research has systematically examined the real-world impacts of LLM availability on educational equity beyond theoretical projections and controlled studies of innovative LLM applications. To depict trends of post-LLM inequalities, we analyze 1,140,328 academic writing submissions from 16,791 college students across 2,391 courses between 2021 and 2024 at a public, minority-serving institution in the US. We find that students' overall writing quality gradually increased following the availability of LLMs and that the writing quality gaps between linguistically advantaged and disadvantaged students became increasingly narrower. However, this equitizing effect was more concentrated on students with higher socioeconomic status. These findings shed light on the digital divides in the era of LLMs and raise questions about the equity benefits of LLMs in early stages and highlight the need for researchers and practitioners on developing responsible practices to improve educational equity through LLMs.