From Coarse to Fine: Benchmarking and Reward Modeling for Writing-Centric Generation Tasks

TL;DR

This paper introduces WEval and WRL, a fine-grained evaluation and training framework for writing reward models, significantly improving performance on writing benchmarks and requirement adherence.

Contribution

It presents a novel fine-grained evaluation pipeline and reinforcement learning framework for writing reward models, addressing limitations of existing coarse methods.

Findings

Models show substantial improvements across various writing benchmarks.

The evaluation correlates well with gold rankings, indicating accurate assessment.

Models generalize strongly to different writing requirements.

Abstract

Large language models have achieved remarkable progress in text generation but still struggle with generative writing tasks. In terms of evaluation, existing benchmarks evaluate writing reward models coarsely and fail to measure performance from the perspective of specific requirements. In terms of training, existing training methods either use LLM-as-a-judge approaches or train coarse-grained reward models, lacking fine-grained requirement-adherence reward modeling. To address these issues, we propose a fine-grained evaluation pipeline WEval for writing reward models and a fine-grained reinforcement learning training framework WRL. The evaluation data of WEval covers multiple task categories and requirement types, enabling systematic evaluation of writing reward models by measuring the correlation between the rankings of the reward model and gold rankings. WRL constructs positive and negative samples by selectively dropping instruction requirements, allowing for more precise reward model training. Experiments show that our models achieve substantial improvements across various writing benchmarks and exhibit strong generalization. The code and data are publicly available at \href{https://github.com/Rainier-rq1/From_Coarse_to_Fine}{https://github.com/Rainier-rq1/From\_Coarse\_to\_Fine}.