Efficient Alignment of Large Language Models via Data Sampling

TL;DR

This paper investigates data sampling strategies for large language model alignment, revealing an exponential plateau in performance and proposing an information theory-based method that achieves high-quality alignment with significantly reduced data and resources.

Contribution

It introduces a novel data subsampling approach based on information theory, enabling efficient LLM alignment with less than 10% of the data used in traditional methods.

Findings

Alignment performance follows an exponential plateau pattern.

The proposed method outperforms other sampling techniques.

Achieves over 90% savings in resources while maintaining quality.

Abstract

LLM alignment ensures that large language models behave safely and effectively by aligning their outputs with human values, goals, and intentions. Aligning LLMs employ huge amounts of data, computation, and time. Moreover, curating data with human feedback is expensive and takes time. Recent research depicts the benefit of data engineering in the fine-tuning and pre-training paradigms to bring down such costs. However, alignment differs from the afore-mentioned paradigms and it is unclear if data efficient alignment is feasible. In this work, we first aim to understand how the performance of LLM alignment scales with data. We find out that LLM alignment performance follows an exponential plateau pattern which tapers off post a rapid initial increase. Based on this, we identify data subsampling as a viable method to reduce resources required for alignment. Further, we propose an information theory-based methodology for efficient alignment by identifying a small high quality subset thereby reducing the computation and time required by alignment. We evaluate the proposed methodology over multiple datasets and compare the results. We find that the model aligned using our proposed methodology outperforms other sampling methods and performs comparable to the model aligned with the full dataset while using less than 10% data, leading to greater than 90% savings in costs, resources, and faster LLM alignment.