A Corpus for Modeling Word Importance in Spoken Dialogue Transcripts

TL;DR

This paper introduces a new annotated corpus of spoken dialogue with word importance scores, trains a model to predict these scores, and discusses applications for improving speech recognition for the deaf and hard-of-hearing.

Contribution

It provides a novel corpus with word importance annotations and a trained model for automatic importance prediction in spoken dialogue transcripts.

Findings

Best model achieves 0.60 F-score in 6-class importance classification

Model agreement with human annotators has a CCC of 0.839

Corpus enables better evaluation of ASR for DHH users

Abstract

Motivated by a project to create a system for people who are deaf or hard-of-hearing that would use automatic speech recognition (ASR) to produce real-time text captions of spoken English during in-person meetings with hearing individuals, we have augmented a transcript of the Switchboard conversational dialogue corpus with an overlay of word-importance annotations, with a numeric score for each word, to indicate its importance to the meaning of each dialogue turn. Further, we demonstrate the utility of this corpus by training an automatic word importance labeling model; our best performing model has an F-score of 0.60 in an ordinal 6-class word-importance classification task with an agreement (concordance correlation coefficient) of 0.839 with the human annotators (agreement score between annotators is 0.89). Finally, we discuss our intended future applications of this resource, particularly for the task of evaluating ASR performance, i.e. creating metrics that predict ASR-output caption text usability for DHH users better thanWord Error Rate (WER).