Toward Effective Secure Code Reviews: An Empirical Study of Security-Related Coding Weaknesses

TL;DR

This empirical study analyzes how security-related coding weaknesses are identified and addressed during code reviews in open-source projects, revealing gaps in current practices and suggesting the need for improved security awareness.

Contribution

It provides the first large-scale empirical analysis of security-related coding weaknesses discussed during code reviews, highlighting current shortcomings and areas for improvement.

Findings

Reviewers raised security concerns in 35 out of 40 weakness categories.

Many security concerns were acknowledged but not fixed, indicating gaps in fixing security issues.

Some weaknesses related to past vulnerabilities were discussed less often than expected.

Abstract

Identifying security issues early is encouraged to reduce the latent negative impacts on software systems. Code review is a widely-used method that allows developers to manually inspect modified code, catching security issues during a software development cycle. However, existing code review studies often focus on known vulnerabilities, neglecting coding weaknesses, which can introduce real-world security issues that are more visible through code review. The practices of code reviews in identifying such coding weaknesses are not yet fully investigated. To better understand this, we conducted an empirical case study in two large open-source projects, OpenSSL and PHP. Based on 135,560 code review comments, we found that reviewers raised security concerns in 35 out of 40 coding weakness categories. Surprisingly, some coding weaknesses related to past vulnerabilities, such as memory errors and resource management, were discussed less often than the vulnerabilities. Developers attempted to address raised security concerns in many cases (39%-41%), but a substantial portion was merely acknowledged (30%-36%), and some went unfixed due to disagreements about solutions (18%-20%). This highlights that coding weaknesses can slip through code review even when identified. Our findings suggest that reviewers can identify various coding weaknesses leading to security issues during code reviews. However, these results also reveal shortcomings in current code review practices, indicating the need for more effective mechanisms or support for increasing awareness of security issue management in code reviews.