Decoding AI Tutor Effects for Educational Measurement: Temporal, Multi-Outcome, and Behavior-Cognitive Analysis

TL;DR

This study develops an AI tutor framework to analyze how early interactions predict future performance and trust, identify learner profiles, and examine multi-outcome trade-offs in AI-assisted learning.

Contribution

It introduces a novel AI agent prototype with a neural policy model and simulation framework for comprehensive temporal, multi-outcome, and behavioral analysis.

Findings

Early interaction patterns predict later performance and trust.

Learner behavior evolves over time with AI tutoring.

Latent learner profiles can be identified based on behavioral and cognitive data.

Abstract

Artificial intelligence (AI) tutors have become increasingly popular in learning environments. In this study, we propose an AI agent prototype framework for exploring AI-assisted learning with temporal interaction patterns, multiple outcomes analysis, and behavioral-cognitive learner profiling. Based on three research questions, this study aims to investigate whether early interaction patterns can predict later performance and trust, how multiple outcomes can be traded off with different AI tutor feedback conditions, and if learner profiles can be identified with behavioral and cognitive indicators. An AI tutor agent has been developed to provide various feedback forms to learners, including hints, explanations, examples, and code. A neural policy model and a stochastic simulation framework are used to produce artificial student-AI tutor interaction records, which include response time, attempts, hint requests, correctness, quiz results, improvement, satisfaction, and trust. Temporal features are used to predict later correctness and trust with early interaction patterns, and clustering methods are used to find learner profiles. The results showed that early interaction patterns were predictive of later performance and trust, that student behavior changed over time with AI-based tutoring, and that latent student profiles could be identified based on their behavioral and cognitive differences.