Enhancing Critical Thinking in Generative AI Search with Metacognitive Prompts

TL;DR

This study investigates how metacognitive prompts can enhance critical thinking and active engagement in users during Generative AI search, showing promising results in promoting deeper inquiry and perspective-taking.

Contribution

It introduces and empirically tests the effectiveness of metacognitive prompts in supporting critical thinking in human-AI search interactions.

Findings

Metacognitive prompts increased active engagement and exploration.

Prompts helped users consider overlooked perspectives.

Effectiveness varied with students' metacognitive flexibility.

Abstract

The growing use of Generative AI (GenAI) conversational search tools has raised concerns about their effects on people's metacognitive engagement, critical thinking, and learning. As people increasingly rely on GenAI to perform tasks such as analyzing and applying information, they may become less actively engaged in thinking and learning. This study examines whether metacognitive prompts - designed to encourage people to pause, reflect, assess their understanding, and consider multiple perspectives - can support critical thinking during GenAI-based search. We conducted a user study (N=40) with university students to investigate the impact of metacognitive prompts on their thought processes and search behaviors while searching with a GenAI tool. We found that these prompts led to more active engagement, prompting students to explore a broader range of topics and engage in deeper inquiry through follow-up queries. Students reported that the prompts were especially helpful for considering overlooked perspectives, promoting evaluation of AI responses, and identifying key takeaways. Additionally, the effectiveness of these prompts was influenced by students' metacognitive flexibility. Our findings highlight the potential of metacognitive prompts to foster critical thinking and provide insights for designing and implementing metacognitive support in human-AI interactions.