Demystifying Software Release Note Issues on GitHub

TL;DR

This paper provides an empirical analysis of 1,731 GitHub issues to classify common problems in software release notes, revealing key challenges in content accuracy, presentation, accessibility, and production automation.

Contribution

It introduces a comprehensive taxonomy of release note issues based on empirical data, highlighting areas for improvement and future automation research.

Findings

Nearly half of issues focus on production challenges.

RN producers often miss critical information, especially for breaking changes.

Accessibility issues include link deterioration and lack of notifications.

Abstract

Release notes (RNs) summarize main changes between two consecutive software versions and serve as a central source of information when users upgrade software. While producing high quality RNs can be hard and poses a variety of challenges to developers, a comprehensive empirical understanding of these challenges is still lacking. In this paper, we bridge this knowledge gap by manually analyzing 1,731 latest GitHub issues to build a comprehensive taxonomy of RN issues with four dimensions: Content, Presentation, Accessibility, and Production. Among these issues, nearly half (48.47%) of them focus on Production; Content, Accessibility, and Presentation take 25.61%, 17.65%, and 8.27%, respectively. We find that: 1) RN producers are more likely to miss information than to include incorrect information, especially for breaking changes; 2) improper layout may bury important information and confuse users; 3) many users find RNs inaccessible due to link deterioration, lack of notification, and obfuscate RN locations; 4) automating and regulating RN production remains challenging despite the great needs of RN producers. Our taxonomy not only pictures a roadmap to improve RN production in practice but also reveals interesting future research directions for automating RN production.