Towards A Human-in-the-Loop LLM Approach to Collaborative Discourse Analysis

TL;DR

This paper explores using GPT-4-Turbo with human-in-the-loop prompt engineering to analyze and categorize students' collaborative discourse, aiming to characterize synergistic learning effectively.

Contribution

It introduces a novel human-in-the-loop approach employing GPT-4-Turbo to analyze student discourse, a first step towards automated characterization of synergistic learning.

Findings

GPT-4-Turbo may match human performance in characterizing synergistic learning

The approach shows promise for further research and development

Preliminary results support the feasibility of automated discourse analysis

Abstract

LLMs have demonstrated proficiency in contextualizing their outputs using human input, often matching or beating human-level performance on a variety of tasks. However, LLMs have not yet been used to characterize synergistic learning in students' collaborative discourse. In this exploratory work, we take a first step towards adopting a human-in-the-loop prompt engineering approach with GPT-4-Turbo to summarize and categorize students' synergistic learning during collaborative discourse. Our preliminary findings suggest GPT-4-Turbo may be able to characterize students' synergistic learning in a manner comparable to humans and that our approach warrants further investigation.