DIVE: Diversified Iterative Self-Improvement

TL;DR

DIVE introduces a novel framework for iterative self-improvement in large language models that significantly enhances output diversity while maintaining performance, addressing a key limitation of existing methods.

Contribution

DIVE's key innovation lies in combining sample pool expansion and data selection to improve diversity in self-improvement processes for LLMs.

Findings

Achieves 10-45% increase in diversity metrics

Maintains performance quality comparable to vanilla ISI

Ablation studies confirm the importance of both components

Abstract

Recent advances in large language models (LLMs) have demonstrated the effectiveness of Iterative Self-Improvement (ISI) techniques. However, continuous training on self-generated data leads to reduced output diversity, a limitation particularly critical in reasoning tasks where diverse solution paths are essential. We present DIVE (Diversified Iterative Self-Improvement), a novel framework that addresses this challenge through two key components: Sample Pool Expansion for broader solution exploration, and Data Selection for balancing diversity and quality in preference pairs. Experiments on MATH and GSM8k datasets show that DIVE achieves a 10% to 45% relative increase in output diversity metrics while maintaining performance quality compared to vanilla ISI. Our ablation studies confirm both components' significance in achieving these improvements. Code is available at https://github.com/qinyiwei/DIVE.