Experimental Evidence on Negative Impact of Generative AI on Scientific Learning Outcomes

TL;DR

This study experimentally investigates how reliance on Generative AI affects students' learning outcomes, revealing negative impacts on comprehension accuracy but potential benefits in summarization quality, with implications for educational policies.

Contribution

It provides empirical evidence on the differential effects of AI assistance in reading and writing tasks, highlighting the importance of balanced AI use in education.

Findings

Complete reliance on AI reduces accuracy by 25.1%.

AI-assisted reading causes a 12% decline in comprehension accuracy.

AI summarization improves output quality and benefits skilled individuals.

Abstract

In this study, I explored the impact of Generative AI on learning efficacy in academic reading materials using experimental methods. College-educated participants engaged in three cycles of reading and writing tasks. After each cycle, they responded to comprehension questions related to the material. After adjusting for background knowledge and demographic factors, complete reliance on AI for writing tasks led to a 25.1% reduction in accuracy. In contrast, AI-assisted reading resulted in a 12% decline. Interestingly, using AI for summarization significantly improved both quality and output. Accuracy exhibited notable variance in the AI-assisted section. Further analysis revealed that individuals with a robust background in the reading topic and superior reading/writing skills benefitted the most. I conclude the research by discussing educational policy implications, emphasizing the need for educators to warn students about the dangers of over-dependence on AI and provide guidance on its optimal use in educational settings.