Learn What NOT to Learn: Towards Generative Safety in Chatbots

TL;DR

This paper introduces LOT, a contrastive learning framework that improves chatbot safety by reducing toxic outputs without compromising conversational quality, using learned safe and unsafe language distributions.

Contribution

The paper presents a novel contrastive learning approach that automatically leverages safe and unsafe language signals to enhance safety in generative chatbots while maintaining engagement.

Findings

Reduces toxicity by up to four times

Increases engagingness and fluency four to six times

Human evaluation confirms effectiveness

Abstract

Conversational models that are generative and open-domain are particularly susceptible to generating unsafe content since they are trained on web-based social data. Prior approaches to mitigating this issue have drawbacks, such as disrupting the flow of conversation, limited generalization to unseen toxic input contexts, and sacrificing the quality of the dialogue for the sake of safety. In this paper, we present a novel framework, named "LOT" (Learn NOT to), that employs a contrastive loss to enhance generalization by learning from both positive and negative training signals. Our approach differs from the standard contrastive learning framework in that it automatically obtains positive and negative signals from the safe and unsafe language distributions that have been learned beforehand. The LOT framework utilizes divergence to steer the generations away from the unsafe subspace and towards the safe subspace while sustaining the flow of conversation. Our approach is memory and time-efficient during decoding and effectively reduces toxicity while preserving engagingness and fluency. Empirical results indicate that LOT reduces toxicity by up to four-fold while achieving four to six-fold higher rates of engagingness and fluency compared to baseline models. Our findings are further corroborated by human evaluation.