Analysing the potential of seq-to-seq models for incremental interpretation in task-oriented dialogue

TL;DR

This study examines how seq-to-seq models handle disfluencies in task-oriented dialogues, revealing they are resilient to disfluencies and that such noise can even improve overall representation clarity.

Contribution

It demonstrates that seq-to-seq models are robust to disfluencies and provides insights into their internal representations and effects of disfluency data augmentation.

Findings

Disfluencies have minimal impact on task success.

Models develop limited awareness of disfluency structure.

Adding disfluencies can improve representation clarity.

Abstract

We investigate how encoder-decoder models trained on a synthetic dataset of task-oriented dialogues process disfluencies, such as hesitations and self-corrections. We find that, contrary to earlier results, disfluencies have very little impact on the task success of seq-to-seq models with attention. Using visualisation and diagnostic classifiers, we analyse the representations that are incrementally built by the model, and discover that models develop little to no awareness of the structure of disfluencies. However, adding disfluencies to the data appears to help the model create clearer representations overall, as evidenced by the attention patterns the different models exhibit.