sDPO: Don't Use Your Data All at Once

TL;DR

sDPO introduces a stepwise preference optimization method for aligning large language models, improving performance and alignment quality by sequentially utilizing preference datasets during training.

Contribution

It extends DPO by implementing a stepwise approach, enabling better alignment and performance with fewer data and more precise reference models.

Findings

sDPO outperforms other LLMs with more parameters.

Stepwise training improves alignment accuracy.

Method enhances the use of preference datasets in LLM tuning.

Abstract

As development of large language models (LLM) progresses, aligning them with human preferences has become increasingly important. We propose stepwise DPO (sDPO), an extension of the recently popularized direct preference optimization (DPO) for alignment tuning. This approach involves dividing the available preference datasets and utilizing them in a stepwise manner, rather than employing it all at once. We demonstrate that this method facilitates the use of more precisely aligned reference models within the DPO training framework. Furthermore, sDPO trains the final model to be more performant, even outperforming other popular LLMs with more parameters.