Just Say No: Analyzing the Stance of Neural Dialogue Generation in Offensive Contexts

TL;DR

This paper investigates the tendency of neural dialogue models to produce offensive responses, introduces a dataset for analysis, and evaluates methods to reduce offensive output, aiming to improve dialogue safety.

Contribution

The paper presents ToxiChat, a new dataset for analyzing offensive language in dialogue, and evaluates controllable text generation techniques to mitigate offensive responses in neural models.

Findings

42% of human responses agree with toxic comments

Neural models like DialoGPT are twice as likely to agree with offensive comments

Fine-tuned classifiers achieve 0.71 F1 on offensive detection

Abstract

Dialogue models trained on human conversations inadvertently learn to generate toxic responses. In addition to producing explicitly offensive utterances, these models can also implicitly insult a group or individual by aligning themselves with an offensive statement. To better understand the dynamics of contextually offensive language, we investigate the stance of dialogue model responses in offensive Reddit conversations. Specifically, we create ToxiChat, a crowd-annotated dataset of 2,000 Reddit threads and model responses labeled with offensive language and stance. Our analysis reveals that 42% of human responses agree with toxic comments, whereas only 13% agree with safe comments. This undesirable behavior is learned by neural dialogue models, such as DialoGPT, which we show are two times more likely to agree with offensive comments. To enable automatic detection of offensive language, we fine-tuned transformer-based classifiers on ToxiChat that achieve 0.71 F1 for offensive labels and 0.53 Macro-F1 for stance labels. Finally, we quantify the effectiveness of controllable text generation (CTG) methods to mitigate the tendency of neural dialogue models to agree with offensive comments. Compared to the baseline, our best CTG model achieves a 19% reduction in agreement with offensive comments and produces 29% fewer offensive replies. Our work highlights the need for further efforts to characterize and analyze inappropriate behavior in dialogue models, in order to help make them safer. Our code and corpus are available at https://github.com/abaheti95/ToxiChat .