Short-Term Gains, Long-Term Gaps: The Impact of GenAI and Search Technologies on Retention

TL;DR

This study examines how GenAI tools like ChatGPT and search engines impact student performance and retention, finding immediate benefits for lower-order tasks but no long-term retention advantage, emphasizing balanced educational strategies.

Contribution

It provides empirical evidence on the short-term and long-term effects of GenAI and search tools on different cognitive tasks in education.

Findings

AI tools improve immediate performance on lower-order tasks

Retention benefits from AI tools diminish over time

Higher-order tasks show no significant performance difference

Abstract

The rise of Generative AI (GenAI) tools, such as ChatGPT, has transformed how students access and engage with information, raising questions about their impact on learning outcomes and retention. This study investigates how GenAI (ChatGPT), search engines (Google), and e-textbooks influence student performance across tasks of varying cognitive complexity, based on Bloom's Taxonomy. Using a sample of 123 students, we examined performance in three tasks: [1] knowing and understanding, [2] applying, and [3] synthesizing, evaluating, and creating. Results indicate that ChatGPT and Google groups outperformed the control group in immediate assessments for lower-order cognitive tasks, benefiting from quick access to structured information. However, their advantage diminished over time, with retention test scores aligning with those of the e-textbook group. For higher-order cognitive tasks, no significant differences were observed among groups, with the control group demonstrating the highest retention. These findings suggest that while AI-driven tools facilitate immediate performance, they do not inherently reinforce long-term retention unless supported by structured learning strategies. The study highlights the need for balanced technology integration in education, ensuring that AI tools are paired with pedagogical approaches that promote deep cognitive engagement and knowledge retention.