Combining Log Data and Collaborative Dialogue Features to Predict Project Quality in Middle School AI Education

TL;DR

This study explores how combining dialogue transcripts and system interaction logs can predict project quality in middle school AI education, revealing modality-specific strengths and the benefits of multimodal fusion for certain outcomes.

Contribution

It demonstrates the nuanced effectiveness of multimodal data fusion in predicting different aspects of project quality in collaborative AI learning environments.

Findings

Log data better predicts productivity.

Dialogue data more effectively predicts content richness.

Fusion improves predictions for productivity and lexical variation.

Abstract

Project-based learning plays a crucial role in computing education. However, its open-ended nature makes tracking project development and assessing success challenging. We investigate how dialogue and system interaction logs predict project quality during collaborative, project-based AI learning of 94 middle school students working in pairs. We used linguistic features from dialogue transcripts and behavioral features from system logs to predict three project quality outcomes: productivity (number of training phrases), content richness (word density), and lexical variation (word diversity) of chatbot training phrases. We compared the predictive accuracy of each modality and a fusion of the modalities. Results indicate log data better predicts productivity, while dialogue data is more effective for content richness. Both modalities modestly predict lexical variation. Multimodal fusion improved predictions for productivity and lexical variation of training phrases but not content richness. These findings suggest that the value of multimodal fusion depends on the specific learning outcome. The study contributes to multimodal learning analytics by demonstrating the nuanced interplay between behavioral and linguistic data in assessing student learning progress in open-ended AI learning environments.