ReleaseEval: A Benchmark for Evaluating Language Models in Automated Release Note Generation

TL;DR

ReleaseEval is a comprehensive, openly licensed benchmark that evaluates language models' effectiveness in automated release note generation across various input granularities, revealing strengths and challenges of current models.

Contribution

The paper introduces ReleaseEval, a large-scale, reproducible benchmark dataset for evaluating language models on automated release note generation tasks with multiple input formats.

Findings

LLMs outperform traditional baselines across tasks

Large gains observed on tree2sum task

Models struggle with diff2sum due to complex code diffs

Abstract

Automated release note generation addresses the challenge of documenting frequent software updates, where manual efforts are time-consuming and prone to human error. Although recent advances in language models further enhance this process, progress remains hindered by dataset limitations, including the lack of explicit licensing and limited reproducibility, and incomplete task design that relies mainly on commit messages for summarization while overlooking fine-grained contexts such as commit hierarchies and code changes. To fill this gap, we introduce ReleaseEval, a reproducible and openly licensed benchmark designed to systematically evaluate language models for automated release note generation. ReleaseEval comprises 94,987 release notes from 3,369 repositories across 6 programming languages, and supports three task settings with three levels of input granularity: (1) commit2sum, which generates release notes from commit messages; (2) tree2sum, which incorporates commit tree structures; and (3) diff2sum, which leverages fine-grained code diffs. Both automated and human evaluations show that large language models consistently outperform traditional baselines across all tasks, achieving substantial gains on tree2sum, while still struggling on diff2sum. These findings highlight LLMs' proficiency in leveraging structured information while revealing challenges in abstracting from long code diffs.