Build It, Break It, Fix It: Contesting Secure Development

TL;DR

The BIBIFI contest evaluates both secure software development and security-breaking skills, revealing that language choice impacts security flaws and that successful builders are better at finding bugs.

Contribution

Introduces a novel contest framework that assesses secure building and breaking of software, providing insights into factors influencing security and bug detection.

Findings

C/C++ submissions were most efficient for building

Statically-typed safe languages reduced security flaws by 11 times

Successful builders excelled at finding security bugs

Abstract

Typical security contests focus on breaking or mitigating the impact of buggy systems. We present the Build-it, Break-it, Fix-it (BIBIFI) contest, which aims to assess the ability to securely build software, not just break it. In BIBIFI, teams build specified software with the goal of maximizing correctness, performance, and security. The latter is tested when teams attempt to break other teams' submissions. Winners are chosen from among the best builders and the best breakers. BIBIFI was designed to be open-ended; teams can use any language, tool, process, etc. that they like. As such, contest outcomes shed light on factors that correlate with successfully building secure software and breaking insecure software. We ran three contests involving a total of 156 teams and three different programming problems. Quantitative analysis from these contests found that the most efficient build-it submissions used C/C++, but submissions coded in a statically-type safe language were 11 times less likely to have a security flaw than C/C++ submissions. Break-it teams that were also successful build-it teams were significantly better at finding security bugs.