Preference Packing: Efficient Preference Optimization for Large Language Models

TL;DR

Preference packing is a novel method that enhances resource efficiency in training large language models by reducing attention operations and memory usage, leading to significant speedups especially when combined with other optimization techniques.

Contribution

It introduces preference packing, a new approach that improves training efficiency for models using data with multiple responses, reducing computational costs and enabling faster training.

Findings

Achieved at least 37% reduction in training time.

Can be combined with batch sorting for a 3.22x speedup.

Effective on both text-only and image-included datasets.

Abstract

Resource-efficient training optimization techniques are becoming increasingly important as the size of large language models (LLMs) continues to grow. In particular, batch packing is commonly used in pre-training and supervised fine-tuning to achieve resource-efficient training. We propose preference packing, a method to enhance resource efficiency in training techniques that use data with different responses for the same input prompt, such as reward models or Direct Preference Optimization (DPO). Preference packing improves resource efficiency by reducing the attention operations for duplicate input prompts and decreasing KV cache memory usage. We conducted experiments on text-only datasets and image-included datasets and achieved at least 37% reduction in training time. Notably, this method can be applied alongside existing optimization techniques such as batch sorting, resulting in a 3.22x speedup.