Demystifying the oracle: A "20 Questions" game to promote AI ethics and literacy

TL;DR

This paper introduces a didactic '20 Questions' game activity to help students understand the probabilistic nature of Large Language Models, promoting AI literacy and ethical awareness in physics education.

Contribution

It presents a novel interactive activity that visualizes LLMs' stochastic behavior, aiding in teaching AI literacy and addressing ethical misconceptions about AI models.

Findings

Students observe divergent responses from LLMs with identical inputs.

The activity enhances understanding of AI's probabilistic nature.

Mapping AI behavior to physics concepts fosters informed skepticism.

Abstract

As Generative AI becomes a key component in physics education, a significant ethical challenge has emerged: the tendency of students to anthropomorphize Large Language Models (LLMs), treating them as authoritative "oracles" that retrieve fixed facts from an internal database. However, LLMs operate fundamentally as probabilistic engines. This paper describes the design and implementation of a didactic activity, a reduced version of the "20 Questions" game, aimed at making this stochastic nature directly observable. Unlike a human player who fixes a target object at the start of the game, students discover that the model generates answers based solely on local coherence with the interaction history. By utilizing functionalities such as re-sampling and history rewinding, students act as experimenters, observing how identical interaction histories can yield diverging narrative paths. We discuss how mapping these behaviors to familiar physics concepts provides the epistemic scaffolding necessary to promote informed skepticism, framing the verification of AI outputs not merely as a compliance rule, but as a technical necessity derived from the system's probabilistic nature.