The CRINGE Loss: Learning what language not to model

TL;DR

The paper introduces the CRINGE loss, a novel training method that incorporates negative examples to improve language models in safety, contradiction avoidance, and dialogue tasks, outperforming existing baselines.

Contribution

It presents a new negative data training procedure called CRINGE loss, enhancing language model safety and reliability with simple, effective implementation.

Findings

CRINGE loss improves safe generation and contradiction avoidance.

Models trained with CRINGE outperform strong baselines.

The approach is easy to train and implement.

Abstract

Standard language model training employs gold human documents or human-human interaction data, and treats all training data as positive examples. Growing evidence shows that even with very large amounts of positive training data, issues remain that can be alleviated with relatively small amounts of negative data -- examples of what the model should not do. In this work, we propose a novel procedure to train with such data called the CRINGE loss (ContRastive Iterative Negative GEneration). We show the effectiveness of this approach across three different experiments on the tasks of safe generation, contradiction avoidance, and open-domain dialogue. Our models outperform multiple strong baselines and are conceptually simple, easy to train and implement.