"AI just keeps guessing": Using ARC Puzzles to Help Children Identify Reasoning Errors in Generative AI

TL;DR

This paper presents AI Puzzlers, an interactive system using ARC puzzles to help children detect and understand reasoning errors in generative AI, addressing challenges of error detection and overtrust in AI outputs.

Contribution

It introduces a novel educational tool based on ARC puzzles that supports children in identifying and analyzing AI errors, informed by cognitive learning theories.

Findings

Children can identify errors in genAI outputs using AI Puzzlers.

The system helps children develop strategies for error detection.

Design insights improve understanding of children's interaction with AI errors.

Abstract

The integration of generative Artificial Intelligence (genAI) into everyday life raises questions about the competencies required to critically engage with these technologies. Unlike visual errors in genAI, textual mistakes are often harder to detect and require specific domain knowledge. Furthermore, AI's authoritative tone and structured responses can create an illusion of correctness, leading to overtrust, especially among children. To address this, we developed AI Puzzlers, an interactive system based on the Abstraction and Reasoning Corpus (ARC), to help children identify and analyze errors in genAI. Drawing on Mayer & Moreno's Cognitive Theory of Multimedia Learning, AI Puzzlers uses visual and verbal elements to reduce cognitive overload and support error detection. Based on two participatory design sessions with 21 children (ages 6 - 11), our findings provide both design insights and an empirical understanding of how children identify errors in genAI reasoning, develop strategies for navigating these errors, and evaluate AI outputs.