Security Defect Detection via Code Review: A Study of the OpenStack and Qt Communities

TL;DR

This study empirically investigates how security defects are identified and addressed through code review in open-source projects, highlighting the complementary role of manual review and automated tools in security defect detection.

Contribution

It provides an empirical analysis of security defect discussions in code reviews, revealing insights into reviewer strategies and developer responses in open-source communities.

Findings

Security defects are infrequently discussed in code reviews.

Over half of reviewers suggest specific fixes for security issues.

Developers tend to follow reviewers' security suggestions.

Abstract

Background: Despite the widespread use of automated security defect detection tools, software projects still contain many security defects that could result in serious damage. Such tools are largely context-insensitive and may not cover all possible scenarios in testing potential issues, which makes them susceptible to missing complex security defects. Hence, thorough detection entails a synergistic cooperation between these tools and human-intensive detection techniques, including code review. Code review is widely recognized as a crucial and effective practice for identifying security defects. Aim: This work aims to empirically investigate security defect detection through code review. Method: To this end, we conducted an empirical study by analyzing code review comments derived from four projects in the OpenStack and Qt communities. Through manually checking 20,995 review comments obtained by keyword-based search, we identified 614 comments as security-related. Results: Our results show that (1) security defects are not prevalently discussed in code review, (2) more than half of the reviewers provided explicit fixing strategies/solutions to help developers fix security defects, (3) developers tend to follow reviewers' suggestions and action the changes, (4) Not worth fixing the defect now and Disagreement between the developer and the reviewer are the main causes for not resolving security defects. Conclusions: Our research results demonstrate that (1) software security practices should combine manual code review with automated detection tools, achieving a more comprehensive coverage to identifying and addressing security defects, and (2) promoting appropriate standardization of practitioners' behaviors during code review remains necessary for enhancing software security.