Comments on Comments: Where Code Review and Documentation Meet

TL;DR

This study analyzes nearly 700,000 code review comments to understand how documentation-related comments influence shared understanding and identify opportunities for automating aspects of code review.

Contribution

It introduces a taxonomy of reviewer intents behind comments on comments and provides insights into their role in clarifying code and documentation issues.

Findings

Clarification is the most common reviewer focus.

Many clarifying comments are suggested verbatim by reviewers.

Identified 65,000 comments likely related to documentation changes.

Abstract

A central function of code review is to increase understanding; helping reviewers understand a code change aids in knowledge transfer and finding bugs. Comments in code largely serve a similar purpose, helping future readers understand the program. It is thus natural to study what happens when these two forms of understanding collide. We ask: what documentation-related comments do reviewers make and how do they affect understanding of the contribution? We analyze ca.700K review comments on 2,000 (Java and Python) GitHub projects, and propose several filters to identify which comments are likely to be either in response to a change in documentation and/or call for such a change. We identify 65K such cases. We next develop a taxonomy of the reviewer intents behind such "comments on comments". We find that achieving a shared understanding of the code is key: reviewer comments most often focused on clarification, followed by pointing out issues to fix, such as typos and outdated comments. Curiously, clarifying comments were frequently suggested (often verbatim) by the reviewer, indicating a desire to persist their understanding acquired during code review. We conclude with a discussion of implications of our comments-on-comments dataset for research on improving code review, including the potential benefits for automating code review.