A User-centered Security Evaluation of Copilot

TL;DR

This study evaluates GitHub's Copilot through a user study to understand its impact on code security, revealing potential security benefits especially for complex problems, but highlighting the need for further research.

Contribution

It provides the first user-centered security evaluation of Copilot, analyzing its effects on developer security performance across different problem complexities.

Findings

Copilot leads to more secure solutions on harder problems.

No significant impact of Copilot on security for easier problems.

Use of Copilot does not disproportionately affect vulnerability types.

Abstract

Code generation tools driven by artificial intelligence have recently become more popular due to advancements in deep learning and natural language processing that have increased their capabilities. The proliferation of these tools may be a double-edged sword because while they can increase developer productivity by making it easier to write code, research has shown that they can also generate insecure code. In this paper, we perform a user-centered evaluation GitHub's Copilot to better understand its strengths and weaknesses with respect to code security. We conduct a user study where participants solve programming problems (with and without Copilot assistance) that have potentially vulnerable solutions. The main goal of the user study is to determine how the use of Copilot affects participants' security performance. In our set of participants (n=25), we find that access to Copilot accompanies a more secure solution when tackling harder problems. For the easier problem, we observe no effect of Copilot access on the security of solutions. We also observe no disproportionate impact of Copilot use on particular kinds of vulnerabilities. Our results indicate that there are potential security benefits to using Copilot, but more research is warranted on the effects of the use of code generation tools on technically complex problems with security requirements.