Thematic and Task-Based Categorization of K-12 GenAI Usages with Hierarchical Topic Modeling

TL;DR

This paper introduces a hierarchical topic modeling approach to categorize K-12 GenAI interactions by content and task, providing insights into usage patterns and applications in educational settings.

Contribution

It presents a novel, simple hierarchical topic modeling method for categorizing K-12 GenAI interactions, supported by real-world data, improving upon prior approaches.

Findings

Hierarchical categorization reveals diverse GenAI usage patterns.

State-of-the-art LLMs outperform traditional topic models in this context.

Insights support educational practices and highlight future research directions.

Abstract

We analyze anonymous interaction data of minors in class-rooms spanning several months, schools, and subjects employing a novel, simple topic modeling approach. Specifically, we categorize more than 17,000 messages generated by students, teachers, and ChatGPT in two dimensions: content (such as nature and people) and tasks (such as writing and explaining). Our hierarchical categorization done separately for each dimension includes exemplary prompts, and provides both a high-level overview as well as tangible insights. Prior works mostly lack a content or thematic categorization. While task categorizations are more prevalent in education, most have not been supported by real-world data for K-12. In turn, it is not surprising that our analysis yielded a number of novel applications. In deriving these insights, we found that many of the well-established classical and emerging computational methods, i.e., topic modeling, for analysis of large amounts of texts underperform, leading us to directly apply state-of-the-art LLMs with adequate pre-processing to achieve hierarchical topic structures with better human alignment through explicit instructions than prior approaches. Our findings support fellow researchers, teachers and students in enriching the usage of GenAI, while our discussion also highlights a number of concerns and open questions for future research.