Learning from negative feedback, or positive feedback or both

TL;DR

This paper introduces a novel EM-based method that enables learning from positive, negative, or both types of feedback, expanding preference optimization to scenarios with unpaired feedback and demonstrating stable learning from negative feedback alone.

Contribution

It extends EM algorithms to incorporate negative feedback explicitly, allowing effective learning even with only one feedback type, which was not addressed by prior methods.

Findings

Effective learning from negative feedback alone demonstrated

Method outperforms existing approaches in unpaired feedback scenarios

Applicable to language models and decision-making policies

Abstract

Existing preference optimization methods often assume scenarios where paired preference feedback (preferred/positive vs. dis-preferred/negative examples) is available. This requirement limits their applicability in scenarios where only unpaired feedback--for example, either positive or negative--is available. To address this, we introduce a novel approach that decouples learning from positive and negative feedback. This decoupling enables control over the influence of each feedback type and, importantly, allows learning even when only one feedback type is present. A key contribution is demonstrating stable learning from negative feedback alone, a capability not well-addressed by current methods. Our approach builds upon the probabilistic framework introduced in (Dayan and Hinton, 1997), which uses expectation-maximization (EM) to directly optimize the probability of positive outcomes (as opposed to classic expected reward maximization). We address a key limitation in current EM-based methods: they solely maximize the likelihood of positive examples, while neglecting negative ones. We show how to extend EM algorithms to explicitly incorporate negative examples, leading to a theoretically grounded algorithm that offers an intuitive and versatile way to learn from both positive and negative feedback. We evaluate our approach for training language models based on human feedback as well as training policies for sequential decision-making problems, where learned value functions are available.