AutoReply: Detecting Nonsense in Dialogue Introspectively with Discriminative Replies

TL;DR

AutoReply introduces an introspective method for dialogue models to detect nonsensical messages by predicting likely responses, eliminating the need for external classifiers, and demonstrating effectiveness in complex game dialogues.

Contribution

The paper presents AutoReply, an automatic algorithm to generate discriminative replies for detecting nonsense in dialogues without external classifiers, outperforming handcrafted replies.

Findings

AutoReply-generated replies outperform handcrafted ones.

Single replies can effectively detect dialogue nonsense.

Method performs comparably to fine-tuned large models.

Abstract

Existing approaches built separate classifiers to detect nonsense in dialogues. In this paper, we show that without external classifiers, dialogue models can detect errors in their own messages introspectively, by calculating the likelihood of replies that are indicative of poor messages. For example, if an agent believes its partner is likely to respond "I don't understand" to a candidate message, that message may not make sense, so an alternative message should be chosen. We evaluate our approach on a dataset from the game Diplomacy, which contains long dialogues richly grounded in the game state, on which existing models make many errors. We first show that hand-crafted replies can be effective for the task of detecting nonsense in applications as complex as Diplomacy. We then design AutoReply, an algorithm to search for such discriminative replies automatically, given a small number of annotated dialogue examples. We find that AutoReply-generated replies outperform handcrafted replies and perform on par with carefully fine-tuned large supervised models. Results also show that one single reply without much computation overheads can also detect dialogue nonsense reasonably well.