Towards an optimised evaluation of teachers' discourse: The case of engaging messages

TL;DR

This study develops a new methodology using large language models to efficiently evaluate and analyze teachers' engaging messages in classroom discourse, aiming to improve teaching quality and student outcomes.

Contribution

Introduces a novel LLM-based approach for assessing teacher discourse, specifically engaging messages, reducing laborious coding and enabling detailed analysis across educational levels.

Findings

Teachers mainly use engagement-promoting messages linked to positive outcomes.

One-third of messages highlight non-engagement disadvantages, increasing student anxiety.

Engaging message use declines in Grade 12 and later in the academic year.

Abstract

Evaluating teachers' skills is crucial for enhancing education quality and student outcomes. Teacher discourse, significantly influencing student performance, is a key component. However, coding this discourse can be laborious. This study addresses this issue by introducing a new methodology for optimising the assessment of teacher discourse. The research consisted of two studies, both within the framework of engaging messages used by secondary education teachers. The first study involved training two large language models on real-world examples from audio-recorded lessons over two academic years to identify and classify the engaging messages from the lessons' transcripts. This resulted in sensitivities of 84.31% and 91.11%, and specificities of 97.69% and 86.36% in identification and classification, respectively. The second study applied these models to transcripts of audio-recorded lessons from a third academic year to examine the frequency and distribution of message types by educational level and moment of the academic year. Results showed teachers predominantly use messages emphasising engagement benefits, linked to improved outcomes, while one-third highlighted non-engagement disadvantages, associated with increased anxiety. The use of engaging messages declined in Grade 12 and towards the academic year's end. These findings suggest potential interventions to optimise engaging message use, enhancing teaching quality and student outcomes.