Exploring Effective Strategies for Building a User-Configured GPT for Coding Classroom Dialogues

TL;DR

This paper explores strategies for customizing GPT-4 to assist in coding classroom dialogues, aiming to improve dialogue analysis with limited data and tailored coding schemes.

Contribution

It introduces practical strategies for configuring a custom GPT for dialogue coding, focusing on small datasets and user-specific coding schemes.

Findings

Custom GPT can generate useful coding suggestions despite limitations.

Performance varies with different example inputs under controlled conditions.

Strategies based on MyGPT's features improve coding assistance effectiveness.

Abstract

This study investigated effective strategies for developing a custom GPT to code classroom dialogue. While classroom dialogue is widely recognised as a crucial element of education, its analysis remains challenging due to the need for a nuanced understanding of dialogic functions and the labour-intensive nature of manual transcript coding. Recent advancements in large language models (LLMs) offer promising avenues for automating this process. However, existing studies predominantly focus on training large-scale models or evaluating pre-trained models with fixed codebooks, the outcomes of which are often not applicable, or the methods are not replicable for dialogue researchers working with small datasets or employing customised coding schemes. Using MyGPT - a GPT-4-based customised GPT system configured for dialogue analysis - as a case, this study evaluates its baseline performance in coding classroom dialogue with a human codebook and examines how performance varies with different example inputs under a controlled variable design. Through a design-based research approach, this study explores a set of practical strategies - based upon MyGPT's unique features - for configuring an effective tool with limited data. The findings suggest that, despite a few limitations, a custom GPT developed using these specific strategies can serve as a useful coding assistant by generating coding suggestions.