A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques

TL;DR

This paper systematically investigates how different choices in parameter-efficient preference alignment methods affect large language model performance, providing insights and guidelines for more effective alignment strategies.

Contribution

It offers an extensive empirical study of the effects of datasets, techniques, and models on preference alignment, revealing key trends and unexpected results.

Findings

More informative data improves preference alignment.

Supervised fine-tuning can outperform preference optimization in some cases.

Aligning to specific preferences enhances downstream task performance.

Abstract

Large language models are first pre-trained on trillions of tokens and then instruction-tuned or aligned to specific preferences. While pre-training remains out of reach for most researchers due to the compute required, fine-tuning has become affordable thanks to parameter-efficient methods such as LoRA and QLoRA. Alignment is known to be sensitive to the many factors involved, including the quantity and quality of data, the alignment method, and the adapter rank. However, there has not yet been an extensive study of their effect on downstream performance. To address this gap, we conduct an in-depth investigation of the impact of popular choices for three crucial axes: (i) the alignment dataset (HH-RLHF and BeaverTails), (ii) the alignment technique (SFT and DPO), and (iii) the model (LLaMA-1, Vicuna-v1.3, Mistral-7b, and Mistral-7b-Instruct). Our extensive setup spanning over 300 experiments reveals consistent trends and unexpected findings. We observe how more informative data helps with preference alignment, cases where supervised fine-tuning outperforms preference optimization, and how aligning to a distinct preference boosts performance on downstream tasks. Through our in-depth analyses, we put forward key guidelines to help researchers perform more effective parameter-efficient LLM alignment.