Disfluency Detection using a Bidirectional LSTM

TL;DR

This paper presents a novel disfluency detection method using a Bidirectional LSTM that incorporates pattern match features and ILP constraints, achieving state-of-the-art results on Switchboard.

Contribution

It introduces a BLSTM model with pattern match features and ILP-based constraints for improved disfluency detection performance.

Findings

Achieves state-of-the-art performance on Switchboard disfluency detection

Better detection of non-repetition disfluencies

Model outperforms previous methods in both detection and correction tasks

Abstract

We introduce a new approach for disfluency detection using a Bidirectional Long-Short Term Memory neural network (BLSTM). In addition to the word sequence, the model takes as input pattern match features that were developed to reduce sensitivity to vocabulary size in training, which lead to improved performance over the word sequence alone. The BLSTM takes advantage of explicit repair states in addition to the standard reparandum states. The final output leverages integer linear programming to incorporate constraints of disfluency structure. In experiments on the Switchboard corpus, the model achieves state-of-the-art performance for both the standard disfluency detection task and the correction detection task. Analysis shows that the model has better detection of non-repetition disfluencies, which tend to be much harder to detect.