Expert Preference-based Evaluation of Automated Related Work Generation

TL;DR

This paper introduces GREP, a multi-turn evaluation framework that incorporates expert preferences and detailed criteria to better assess the quality of automated related work generation in scientific writing.

Contribution

The paper presents GREP, a novel evaluation framework that decomposes expert preferences into fine-grained dimensions and uses contrastive examples, improving assessment robustness over standard LLM judges.

Findings

GREP correlates strongly with human expert assessments.

It outperforms standard LLM judges in evaluating related work sections.

State-of-the-art LLMs often fail to meet validation constraints for related work quality.

Abstract

Expert domain writing, such as scientific writing, typically demands extensive domain knowledge. Although large language models (LLMs) show promising potential in this task, evaluating the quality of automatically generated scientific writing is a crucial open issue, as it requires knowledge of domain-specific criteria and the ability to discern expert preferences. Conventional task-agnostic automatic evaluation metrics and LLM-as-a-judge systems, primarily designed for mainstream NLP tasks, are insufficient to grasp expert preferences and domain-specific quality standards. To address this gap and support realistic human-AI collaborative writing, we focus on related work generation, one of the most challenging scientific tasks, as an exemplar. We propose GREP, a multi-turn evaluation framework that integrates classical related work evaluation criteria with expert-specific preferences. Our framework decomposes the evaluation into smaller fine-grained dimensions. This localized evaluation is further augmented with contrastive examples to provide detailed contextual guidance for the evaluation dimensions. Empirical investigation reveals that our framework is able to assess the quality of related work sections in a much more robust manner compared to standard LLM judges, reflects natural scenarios of scientific writing, and bears a strong correlation with the assessment of human experts. We also observe that generations from state-of-the-art (SoTA) LLMs struggle to satisfy validation constraints of a suitable related work section.