LongDPO: Unlock Better Long-form Generation Abilities for LLMs via Critique-augmented Stepwise Information

TL;DR

This paper introduces LongDPO, a method that enhances long-form generation in large language models by using critique-augmented, stepwise preference learning with Monte Carlo Tree Search and external critiques, leading to improved length and quality.

Contribution

It proposes a novel process supervision approach with Monte Carlo Tree Search and external critiques for better long-form generation in LLMs, outperforming existing methods.

Findings

Improved length and quality in long-form generation benchmarks.

Almost lossless performance on general benchmarks across various models.

Effective use of stepwise preference pairs and external critiques.

Abstract

Long-form generation is crucial for academic writing papers and repo-level code generation. Despite this, current models, including GPT-4o, still exhibit unsatisfactory performance. Existing methods that utilize preference learning with outcome supervision often fail to provide detailed feedback for extended contexts. This shortcoming can lead to content that does not fully satisfy query requirements, resulting in issues like length deviations, and diminished quality. In this paper, we propose enhancing long-form generation by incorporating process supervision. We employ Monte Carlo Tree Search to gather stepwise preference pairs, utilizing a global memory pool to maintain consistency. To address the issue of suboptimal candidate selection, we integrate external critiques to refine and improve the quality of the preference pairs. Finally, we apply step-level DPO using the collected stepwise preference pairs. Experimental results show that our method improves length and quality on long-form generation benchmarks, with almost lossless performance on general benchmarks across various model backbones.