How to Assess AI Literacy: Misalignment Between Self-Reported and Objective-Based Measures

TL;DR

This study develops and evaluates self-report and objective measures of AI literacy among K-12 teachers, revealing discrepancies and profiles that inform scalable professional development and learning analytics.

Contribution

It introduces validated SR and OB measures within an AI literacy framework and analyzes their relationship, providing tools for diagnostics and tailored interventions.

Findings

Low correlation between self-report and objective measures.

Six distinct teacher profiles identified, including overestimation and underestimation.

Measures support scalable learning analytics and professional development.

Abstract

The widespread adoption of Artificial Intelligence (AI) in K-12 education highlights the need for psychometrically-tested measures of teachers' AI literacy. Existing work has primarily relied on either self-report (SR) or objective-based (OB) assessments, with few studies aligning the two within a shared framework to compare perceived versus demonstrated competencies or examine how prior AI literacy experience shapes this relationship. This gap limits the scalability of learning analytics and the development of learner profile-driven instructional design. In this study, we developed and evaluated SR and OB measures of teacher AI literacy within the established framework of Concept, Use, Evaluate, and Ethics. Confirmatory factor analyses support construct validity with good reliability and acceptable fit. Results reveal a low correlation between SR and OB factors. Latent profile analysis identified six distinct profiles, including overestimation (SR > OB), underestimation (SR < OB), alignment (SR close to OB), and a unique low-SR/low-OB profile among teachers without AI literacy experience. Theoretically, this work extends existing AI literacy frameworks by validating SR and OB measures on shared dimensions. Practically, the instruments function as diagnostic tools for professional development, supporting AI-informed decisions (e.g., growth monitoring, needs profiling) and enabling scalable learning analytics interventions tailored to teacher subgroups.