Plug-and-Play Training Framework for Preference Optimization

TL;DR

This paper introduces a flexible training framework that enhances preference optimization for large language models by weighting samples based on difficulty, leading to improved performance in complex tasks like mathematical reasoning.

Contribution

It proposes a novel plug-and-play training method that dynamically weights training samples, addressing sample difficulty and improving preference optimization effectiveness.

Findings

Improved accuracy in mathematical reasoning tasks.

Seamless integration with existing preference optimization methods.

Enhanced learning efficiency for large language models.

Abstract

Recently, preference optimization methods such as DPO have significantly enhanced large language models (LLMs) in wide tasks including dialogue and question-answering. However, current methods fail to account for the varying difficulty levels of training samples during preference optimization, leading to mediocre performance in tasks with high accuracy requirements, particularly in mathematical reasoning. To address this limitation, we propose a novel training framework, which employs multiple sampling to analyze output distributions, assign different weights to samples, and incorporate these weights into the preference optimization process. This plug-and-play approach enables LLMs to prioritize challenging examples during training, improving learning efficiency. Experimental results demonstrate that our framework integrates seamlessly with various preference optimization methods and achieves consistent improvements in mathematical reasoning tasks.