Predicting States of Understanding in Explanatory Interactions Using Cognitive Load-Related Linguistic Cues

TL;DR

This study explores how linguistic cues related to cognitive load can predict a listener's understanding during explanatory dialogues, using multimodal data and machine learning models.

Contribution

It introduces a method combining verbal and nonverbal cues to accurately predict listener understanding states in real-time interactions.

Findings

Individual linguistic cues correlate with understanding levels.

Multimodal classifiers outperform text-only models.

Prediction accuracy improves with combined cues.

Abstract

We investigate how verbal and nonverbal linguistic features, exhibited by speakers and listeners in dialogue, can contribute to predicting the listener's state of understanding in explanatory interactions on a moment-by-moment basis. Specifically, we examine three linguistic cues related to cognitive load and hypothesised to correlate with listener understanding: the information value (operationalised with surprisal) and syntactic complexity of the speaker's utterances, and the variation in the listener's interactive gaze behaviour. Based on statistical analyses of the MUNDEX corpus of face-to-face dialogic board game explanations, we find that individual cues vary with the listener's level of understanding. Listener states ('Understanding', 'Partial Understanding', 'Non-Understanding' and 'Misunderstanding') were self-annotated by the listeners using a retrospective video-recall method. The results of a subsequent classification experiment, involving two off-the-shelf classifiers and a fine-tuned German BERT-based multimodal classifier, demonstrate that prediction of these four states of understanding is generally possible and improves when the three linguistic cues are considered alongside textual features.