LIONs: An Empirically Optimized Approach to Align Language Models

TL;DR

This paper systematically analyzes and optimizes the training pipeline for aligning language models, demonstrating significant performance improvements through specific techniques and training strategies.

Contribution

It provides a comprehensive study of training choices and introduces empirically optimized methods that outperform existing instruct models.

Findings

Sequence packing and loss masking improve fine-tuning.

Increasing preference dataset size enhances alignment.

Online DPO training yields better model performance.

Abstract

Alignment is a crucial step to enhance the instruction-following and conversational abilities of language models. Despite many recent work proposing new algorithms, datasets, and training pipelines, there is a lack of comprehensive studies measuring the impact of various design choices throughout the whole training process. We first conduct a rigorous analysis over a three-stage training pipeline consisting of supervised fine-tuning, offline preference learning, and online preference learning. We have found that using techniques like sequence packing, loss masking in SFT, increasing the preference dataset size in DPO, and online DPO training can significantly improve the performance of language models. We then train from Gemma-2b-base and LLama-3-8b-base, and find that our best models exceed the performance of the official instruct models tuned with closed-source data and algorithms. Our code and models can be found at \url{https://github.com/Columbia-NLP-Lab/LionAlignment}.