Interaction Matters: An Evaluation Framework for Interactive Dialogue Assessment on English Second Language Conversations

TL;DR

This paper introduces an evaluation framework for ESL dialogue interactions that combines interactivity labels and micro-level linguistic features, enabling better assessment of conversation quality.

Contribution

It presents a novel framework integrating micro-level features and interactivity labels for ESL dialogue assessment, with machine learning models revealing key linguistic signals.

Findings

Certain micro-level features strongly correlate with dialogue quality

Reference words like 'she', 'her', 'he' are significant indicators

The framework improves ESL communication assessment

Abstract

We present an evaluation framework for interactive dialogue assessment in the context of English as a Second Language (ESL) speakers. Our framework collects dialogue-level interactivity labels (e.g., topic management; 4 labels in total) and micro-level span features (e.g., backchannels; 17 features in total). Given our annotated data, we study how the micro-level features influence the (higher level) interactivity quality of ESL dialogues by constructing various machine learning-based models. Our results demonstrate that certain micro-level features strongly correlate with interactivity quality, like reference word (e.g., she, her, he), revealing new insights about the interaction between higher-level dialogue quality and lower-level linguistic signals. Our framework also provides a means to assess ESL communication, which is useful for language assessment.