Sample-efficient LLM Optimization with Reset Replay

TL;DR

LoRR is a novel plugin that enhances sample efficiency and mitigates overfitting in preference-based LLM optimization through reset replay and hybrid objectives, improving reasoning benchmarks.

Contribution

Introduces LoRR, a plugin with reset replay and hybrid optimization to boost sample efficiency and reduce overfitting in preference-based LLM training.

Findings

LoRR significantly improves performance on reasoning benchmarks.

An iterative DPO with LoRR rivals complex baselines.

LoRR enables effective learning from limited offline data.

Abstract

Recent advancements in LLM post-training, particularly through reinforcement learning and preference optimization, are key to boosting their reasoning capabilities. However, these methods often suffer from low sample efficiency and a susceptibility to primacy bias, a phenomenon where overfitting to initial experiences diminishes network plasticity and damages the learning process. To address these challenges, we introduce LLM optimization with Reset Replay (LoRR), a general and powerful plugin for enhancing sample efficiency in preference-based optimization. Its core mechanism enables high-replay training to maximize the utility of each data batch. To mitigate overfitting, LoRR orchestrates a periodic reset strategy that reuses the initial data and policy to maintain network plasticity, and further adopts a hybrid optimization objective to better exploit training data. Extensive experiments show that LoRR significantly boosts the performance of various preference optimization methods on both mathematical and general reasoning benchmarks. Notably, an iterative DPO framework augmented with LoRR achieves comparable performance on challenging math tasks, rivaling many complex or computationally expensive baselines. Our findings highlight that LoRR offers a practical and sample-efficient paradigm from limited offline data, unlocking greater performance with minimal changes to existing post-training workflows.