Given Enough Eyeballs, All Bugs Are Shallow? Revisiting Eric Raymond with Bug Bounty Programs

TL;DR

This paper investigates the incentives and dynamics of bug bounty programs, revealing how they influence bug discovery rates and researcher behavior, with implications for improving program design and security outcomes.

Contribution

It provides an empirical analysis of bug bounty program incentives, highlighting the tension between crowd size, bug discovery decay, and researcher switching behavior.

Findings

Researchers tend to switch to new programs with more low-hanging bugs.

Bug discovery probability decays rapidly within programs.

Large crowds increase competition but do not necessarily lead to more bugs found.

Abstract

Bug bounty programs offer a modern platform for organizations to crowdsource their software security and for security researchers to be fairly rewarded for the vulnerabilities they find. Little is known however on the incentives set by bug bounty programs: How they drive new bug discoveries, and how they supposedly improve security through the progressive exhaustion of discoverable vulnerabilities. Here, we recognize that bug bounty programs create tensions, for organizations running them on the one hand, and for security researchers on the other hand. At the level of one bug bounty program, security researchers face a sort of St-Petersburg paradox: The probability of finding additional bugs decays fast, and thus can hardly be matched with a sufficient increase of monetary rewards. Furthermore, bug bounty program managers have an incentive to gather the largest possible crowd to ensure a larger pool of expertise, which in turn increases competition among security researchers. As a result, we find that researchers have high incentives to switch to newly launched programs, for which a reserve of low-hanging fruit vulnerabilities is still available. Our results inform on the technical and economic mechanisms underlying the dynamics of bug bounty program contributions, and may in turn help improve the mechanism design of bug bounty programs that get increasingly adopted by cybersecurity savvy organizations.