Exploring Recurrent, Memory and Attention Based Architectures for Scoring Interactional Aspects of Human-Machine Text Dialog

TL;DR

This paper investigates neural architectures like recurrent, attention, and memory models, combined with feature-engineered methods, for automated scoring of interactional and topic development skills in human-machine text dialogues, aiming to improve feedback for English learners.

Contribution

It compares multiple neural architectures and feature-engineered models for scoring conversational skills, highlighting the effectiveness of transformer-based models in this context.

Findings

Fusion of architectures performs well compared to expert agreement.

Transformer-based models are most effective in scoring.

Feature-engineered models with SVM also contribute significantly.

Abstract

An important step towards enabling English language learners to improve their conversational speaking proficiency involves automated scoring of multiple aspects of interactional competence and subsequent targeted feedback. This paper builds on previous work in this direction to investigate multiple neural architectures -- recurrent, attention and memory based -- along with feature-engineered models for the automated scoring of interactional and topic development aspects of text dialog data. We conducted experiments on a conversational database of text dialogs from human learners interacting with a cloud-based dialog system, which were triple-scored along multiple dimensions of conversational proficiency. We find that fusion of multiple architectures performs competently on our automated scoring task relative to expert inter-rater agreements, with (i) hand-engineered features passed to a support vector learner and (ii) transformer-based architectures contributing most prominently to the fusion.