An Interpretable and Crosslingual Method for Evaluating Second-Language Dialogues

TL;DR

This paper introduces a crosslingual, interpretable framework for evaluating second-language dialogues, demonstrating robustness across English and Chinese, and providing insights into linguistic features influencing dialogue quality.

Contribution

It develops CNIMA, a Chinese second-language dialogue dataset, and proposes a low-data, interpretable evaluation method adaptable to multiple languages.

Findings

Framework is robust across English and Chinese dialogues.

The method reveals language-specific and universal linguistic features.

It does not require labeled data for scoring dialogue quality.

Abstract

We analyse the cross-lingual transferability of a dialogue evaluation framework that assesses the relationships between micro-level linguistic features (e.g. backchannels) and macro-level interactivity labels (e.g. topic management), originally designed for English-as-a-second-language dialogues. To this end, we develop CNIMA (Chinese Non-Native Interactivity Measurement and Automation), a Chinese-as-a-second-language labelled dataset with 10K dialogues. We found the evaluation framework to be robust across distinct languages: English and Chinese, revealing language-specific and language-universal relationships between micro-level and macro-level features. Next, we propose an automated, interpretable approach with low data requirement that scores the overall quality of a second-language dialogue based on the framework. Our approach is interpretable in that it reveals the key linguistic and interactivity features that contributed to the overall quality score. As our approach does not require labelled data, it can also be adapted to other languages for second-language dialogue evaluation.