Language Alignment via Nash-learning and Adaptive feedback

TL;DR

This paper introduces LANA, a novel self-alignment algorithm for large language models that uses Nash-learning and adaptive feedback, eliminating the need for preference datasets and improving alignment through a mirror descent approach.

Contribution

The paper proposes LANA, a new self-alignment method that leverages Nash-learning and adaptive feedback, removing the dependence on preference models or annotated datasets.

Findings

LANA effectively aligns language models without preference datasets.

Experimental results demonstrate improved alignment performance.

Mathematical analysis supports the convergence of the proposed method.

Abstract

Recent research has shown the potential of Nash Learning via Human Feedback for large language model alignment by incorporating the notion of a preference model in a minimax game setup. We take this idea further by casting the alignment as a mirror descent algorithm against the adaptive feedback of an improved opponent, thereby removing the need for learning a preference model or the existence of an annotated dataset altogether. The resulting algorithm, which we refer to as Language Alignment via Nash-learning and Adaptive feedback (LANA), is capable of self-alignment without the need for a human-annotated preference dataset. We support this statement with various experiments and mathematical discussion.