A Systematically Empirical Evaluation of Vulnerability Discovery Models: a Study on Browsers' Vulnerabilities

TL;DR

This paper introduces a systematic empirical methodology to evaluate vulnerability discovery models (VDMs), applying it to browsers' vulnerabilities to assess their quality and predictability, revealing which models are most effective at different stages.

Contribution

It provides the first independent, systematic validation of VDMs, addressing biases in previous studies and offering guidance on model selection based on browser age.

Findings

Some models should be rejected based on performance.

Linear models are best when browsers are young.

Logistics models outperform others as browsers mature.

Abstract

A precise vulnerability discovery model (VDM) will provide a useful insight to assess software security, and could be a good prediction instrument for both software vendors and users to understand security trends and plan ahead patching schedule accordingly. Thus far, several models have been proposed and validated. Yet, no systematically independent validation by somebody other than the author exists. Furthermore, there are a number of issues that might bias previous studies in the field. In this work, we fill in the gap by introducing an empirical methodology that systematically evaluates the performance of a VDM in two aspects: quality and predictability. We further apply this methodology to assess existing VDMs. The results show that some models should be rejected outright, while some others might be adequate to capture the discovery process of vulnerabilities. We also consider different usage scenarios of VDMs and find that the simplest linear model is the most appropriate choice in terms of both quality and predictability when browsers are young. Otherwise, logistics-based models are better choices.