TL;DR
This paper introduces ExPO, a simple method that accelerates large language model alignment by amplifying parameter changes, significantly reducing training time while maintaining or improving performance.
Contribution
ExPO leverages a first-order approximation to enhance alignment training efficiency without additional training, outperforming fully trained models with fewer steps.
Findings
ExPO boosts a 20%-trained model to outperform fully trained counterparts.
ExPO improves alignment of open-source LLMs across multiple benchmarks.
The method reduces training overhead while maintaining high performance.
Abstract
Given the high computational cost of preference alignment training of large language models (LLMs), exploring efficient methods to reduce the training overhead remains an important and compelling research problem. Motivated by the observation that alignment training typically involves only small parameter changes without injecting new knowledge into models, we propose a straightforward method called ExPO (model extrapolation) to expedite LLMs' alignment with human preferences. Given a partially-trained model and its initial SFT checkpoint, ExPO improves the implicit optimization objective of alignment training by simply amplifying the parameter change based on a first-order approximation, without any additional training overhead. Through controlled experiments, we demonstrate that ExPO boosts a DPO model trained with only 20% steps to outperform the fully-trained one. Moreover, we show…
Peer Reviews
Decision·ICLR 2025 Conference Withdrawn Submission
+ The important task of efficiency of LLM alignment, and the interesting idea of extrapolating the LLM weights along the direction of the initial model toward the partially-trained one + Experiments are done on multimple open-source LLMs and bechmarks
- Lack of enough theoretical analysis - Some parameters seem difficult to fix (e.g., \alpha) - English should be improved
1.The paper provides theoretical explanation with comprehensive experiment results to show the effectiveness of EXPO. 2. The paper presents thorough experimental results that span various model architectures and alignment techniques, highlighting EXPO’s flexibility and robustness. 3. The results suggest that EXPO could be beneficial, especially for LLMs, as it provides a computationally economical pathway to enhance alignment.
1. The core idea of EXPO seems to be a straightforward extension of existing model merge concept, primarily adjusting the interpolation parameter to a negative value (extrapolation). While the results are promising, this incremental shift from interpolation to extrapolation may be seen as lacking in true innovation. 2. Unlike interpolation, which typically operates within a bounded range [0,1], the extrapolation parameter α in EXPO operates within an open range [0, +∞). this open-ended range can
- The paper studies a method for improving reward-fine-tuned models. This is an important problem in current LLM research. - The paper is well-written and easy to understand. Figure 2 presents a clear indication of how the method works. Section 2 clearly presents the hypothesis, and the method is neatly summarized by equation 2. - The results presented are significant. Table 1 demonstrates significant gains in Win Rate using the proposed method. Table 5 demonstrates that the results extend to ot
1) To me, there seems to be a disconnect between the theory presented in Section 2 and the results presented. The results seem to point towards different conclusions that contradict the theory. a) Line 192-193: Extrapolation strongly improves the results obtained by "DPO 100%". This is not predicted by the theory presented. The theory presented essentially states, "we can partially train an RLHF model, then predict the result of fully training. The resulting predicted model will achieve the ac
1. The paper tackles an important problem—reducing the computational cost of aligning large language models—which is crucial for scaling models efficiently. 2. The explanation of the method is clear and easy to follow, with helpful figures that enhance understanding. 3. The authors show EXPO’s ability to cut down alignment training costs, which supports their claim.
1. **Limited theoretical foundation**: The paper presents an interesting empirical finding with E X PO, but lacks a rigorous theoretical analysis of why it works. The discussion in Section 3.3 on why E X PO can work is somewhat speculative. A more formal theoretical treatment, perhaps drawing connections to optimization theory or analyzing the loss landscape, would strengthen the paper's contribution. 2. **Limited analysis of failure cases**: While some negative results are reported (e.g. for K
Code & Models
- 🤗chujiezheng/zephyr_0.1model· 2 dl2 dl
- 🤗chujiezheng/zephyr_0.2model· 2 dl2 dl
- 🤗chujiezheng/zephyr_0.05model· 1 dl1 dl
- 🤗chujiezheng/zephyr_0.4model· 3 dl3 dl
- 🤗chujiezheng/Starling-LM-7B-alpha-ExPOmodel· 8.0k dl8.0k dl
- 🤗chujiezheng/Starling-LM-7B-beta-ExPOmodel· 8.1k dl· ♡ 28.1k dl♡ 2
- 🤗chujiezheng/zephyr-7b-dpo-full-ExPOmodel· 7 dl7 dl
- 🤗chujiezheng/tulu-2-dpo-7b-ExPOmodel· 6 dl6 dl
- 🤗chujiezheng/tulu-2-dpo-13b-ExPOmodel· 7 dl7 dl
- 🤗chujiezheng/tulu-2-dpo-70b-ExPOmodel· 8.2k dl8.2k dl
Videos
Taxonomy
TopicsNumerical Methods and Algorithms · Reservoir Engineering and Simulation Methods · Geophysics and Gravity Measurements
MethodsDirect Preference Optimization · Shrink and Fine-Tune · ALIGN
