Reducing the Probability of Undesirable Outputs in Language Models Using Probabilistic Inference

TL;DR

This paper introduces RePULSe, a novel reinforcement learning-based training method for language models that reduces undesirable outputs while maintaining high reward levels, improving robustness and alignment.

Contribution

RePULSe is a new training approach that combines learned proposals with RL to better balance reward optimization and output safety.

Findings

RePULSe outperforms standard RL in reducing undesired outputs.

RePULSe maintains higher expected reward than existing methods.

RePULSe enhances adversarial robustness of language models.

Abstract

Reinforcement learning (RL) has become a predominant technique to align language models (LMs) with human preferences or promote outputs which are deemed to be desirable by a given reward function. Standard RL approaches optimize average reward, while methods explicitly focused on reducing the probability of undesired outputs typically come at a cost to average-case performance. To improve this tradeoff, we introduce RePULSe, a new training method that augments the standard RL loss with an additional loss that uses learned proposals to guide sampling low-reward outputs, and then reduces those outputs' probability. We run experiments demonstrating that RePULSe produces a better tradeoff of expected reward versus the probability of undesired outputs and is more adversarially robust, compared to standard RL alignment approaches and alternatives.