Intermediate direct preference optimization

TL;DR

This paper introduces intermediate direct preference optimization (DPO), a novel method that calculates DPO loss at intermediate layers of large language models to improve fine-tuning performance, showing promising results against traditional methods.

Contribution

The paper proposes a new intermediate DPO approach that computes DPO loss at selected layers, enhancing fine-tuning of large language models compared to conventional final-layer DPO.

Findings

Intermediate DPO achieves higher win rates against conventional DPO.

Performance depends on the choice of intermediate layer position.

Intermediate DPO effectively leverages multiple layer logits during training.

Abstract

We propose the intermediate direct preference optimization (DPO) method to calculate the DPO loss at selected intermediate layers as an auxiliary loss for finetuning large language models (LLMs). The conventional DPO method fine-tunes a supervised fine-tuning (SFT) model by calculating the DPO loss using logits from the final layer. In our intermediate DPO approach, DPO losses are calculated using the logits from K-selected intermediate layers and averaged to obtain the intermediate DPO loss. For training the intermediate DPO model, the final loss is obtained by calculating the weighted sum of the DPO and intermediate DPO losses. During inference, the intermediate DPO model decodes using the final layer logits similarly to the conventional DPO model. In experiments using the ultrafeedback dataset, the performance of the intermediate DPO model was evaluated using GPT-4. As a result, the intermediate DPO model trained using the intermediate DPO loss calculated at the 22nd layer of a 32-layer SFT model achieved win rates of 52.5% and 67.5% against the conventional DPO and SFT models, respectively, demonstrating the effectiveness of the proposed method. Furthermore, we report the relationships among the position of the selected intermediate layers, the number of layers, and performance.