"No, they did not": Dialogue response dynamics in pre-trained language models

TL;DR

This paper investigates pre-trained language models' ability to understand dialogue response dynamics, focusing on at-issueness and ellipsis, revealing strengths in embedded clause sensitivity but limitations in full dynamic comprehension.

Contribution

It provides a detailed analysis of how pre-trained models handle at-issue content and ellipsis, highlighting their partial understanding and areas needing improvement.

Findings

Models are sensitive to embedded clause roles.

Models prefer responses targeting main clause content.

Weak understanding of at-issue versus not-at-issue dynamics.

Abstract

A critical component of competence in language is being able to identify relevant components of an utterance and reply appropriately. In this paper we examine the extent of such dialogue response sensitivity in pre-trained language models, conducting a series of experiments with a particular focus on sensitivity to dynamics involving phenomena of at-issueness and ellipsis. We find that models show clear sensitivity to a distinctive role of embedded clauses, and a general preference for responses that target main clause content of prior utterances. However, the results indicate mixed and generally weak trends with respect to capturing the full range of dynamics involved in targeting at-issue versus not-at-issue content. Additionally, models show fundamental limitations in grasp of the dynamics governing ellipsis, and response selections show clear interference from superficial factors that outweigh the influence of principled discourse constraints.