Dynamic Noise Preference Optimization: Self-Improvement of Large Language Models with Self-Synthetic Data

TL;DR

This paper introduces Dynamic Noise Preference Optimization (DNPO), a novel method for improving large language models through self-synthetic data, dynamic sample labeling, and controlled noise injection, leading to continuous performance gains.

Contribution

The paper proposes DNPO, a new approach combining dynamic preference pair construction and trainable noise injection to enhance LLM self-improvement with synthetic data.

Findings

DNPO outperforms existing methods on multiple benchmarks.

Significant 29.4% win-loss rate gap in data quality with Zephyr-7B.

Enables continuous improvement without performance stagnation.

Abstract

Although LLMs have achieved significant success, their reliance on large volumes of human-annotated data has limited their potential for further scaling. In this situation, utilizing self-generated synthetic data has become crucial for fine-tuning LLMs without extensive human annotation. However, current methods often fail to ensure consistent improvements across iterations, with performance stagnating after only minimal updates. To overcome these challenges, we introduce Dynamic Noise Preference Optimization (DNPO), which combines dynamic sample labeling for constructing preference pairs with controlled, trainable noise injection during preference optimization. Our approach effectively prevents stagnation and enables continuous improvement. In experiments with Llama-3.2-3B and Zephyr-7B, DNPO consistently outperforms existing methods across multiple benchmarks. Additionally, with Zephyr-7B, DNPO shows a significant improvement in model-generated data quality, with a 29.4% win-loss rate gap compared to the baseline in GPT-4 evaluations.