Negative-Prompt-driven Alignment for Generative Language Model

TL;DR

This paper introduces NEAT, a novel alignment method for language models that uses negative prompts to explicitly discourage undesirable outputs, improving alignment with human values.

Contribution

NEAT is the first approach to incorporate negative prompts explicitly in the alignment process, guiding models away from harmful responses during training.

Findings

NEAT significantly reduces harmful and biased outputs.

Models trained with NEAT better align with human preferences.

Enhanced safety and ethical compliance in language generation.

Abstract

Large language models have achieved remarkable capabilities, but aligning their outputs with human values and preferences remains a significant challenge. Existing alignment methods primarily focus on positive examples while overlooking the importance of negative responses in guiding models away from undesirable behaviors. For instance, the widely-used alignment datasets reveals a scarcity of explicit negative examples that contradict human values, hindering its ability to discourage harmful or biased outputs during training. To address this limitation, we propose NEAT, i.e., NEgative-prompt-driven AlignmenT, to introduce negative prompts to generate undesirable responses alongside positive examples during the optimization process. NEAT explicitly penalizes the model for producing harmful outputs, guiding it not only toward desirable behaviors but also steering it away from generating undesirable, biased responses. This dual feedback mechanism enables better alignment with human preferences, crucial in contexts where avoiding harm is paramount. Starting from a pre-trained language model, NEAT performs online alignment by incorporating a ranking loss derived from an expanded preference dataset containing both positive and negative examples. Extensive experiments validate NEAT's effectiveness in significantly enhancing language models' alignment with human values and preferences.