The Effectiveness of Security Interventions on GitHub

TL;DR

This study empirically evaluates the impact of GitHub's security interventions on open source repositories, revealing varying effectiveness and providing design recommendations based on data-driven analysis.

Contribution

It offers the first large-scale, data-driven comparison of GitHub's security interventions, identifying effective components and suggesting improvements.

Findings

All interventions significantly improve security

Effect sizes vary greatly among interventions

Design features influence effectiveness

Abstract

In 2017, GitHub was the first online open source platform to show security alerts to its users. It has since introduced further security interventions to help developers improve the security of their open source software. In this study, we investigate and compare the effects of these interventions. This offers a valuable empirical perspective on security interventions in the context of software development, enriching the predominantly qualitative and survey-based literature landscape with substantial data-driven insights. We conduct a time series analysis on security-altering commits covering the entire history of a large-scale sample of over 50,000 GitHub repositories to infer the causal effects of the security alert, security update, and code scanning interventions. Our analysis shows that while all of GitHub's security interventions have a significant positive effect on security, they differ greatly in their effect size. By comparing the design of each intervention, we identify the building blocks that worked well and those that did not. We also provide recommendations on how practitioners can improve the design of their interventions to enhance their effectiveness.