Do I really need all this work to find vulnerabilities? An empirical case study comparing vulnerability detection techniques on a Java application

TL;DR

This empirical study compares four vulnerability detection techniques on a Java web application, highlighting their effectiveness, efficiency, and unique strengths to inform resource allocation decisions.

Contribution

It provides an empirical comparison of manual and automated vulnerability detection techniques, revealing their unique strengths and resource efficiencies in a real-world Java application.

Findings

SAST detected the most vulnerabilities overall.

EMPT uncovered more severe vulnerabilities.

Manual techniques had comparable or better efficiency than automated ones.

Abstract

CONTEXT: Applying vulnerability detection techniques is one of many tasks using the limited resources of a software project. OBJECTIVE: The goal of this research is to assist managers and other decision-makers in making informed choices about the use of software vulnerability detection techniques through an empirical study of the efficiency and effectiveness of four techniques on a Java-based web application. METHOD: We apply four different categories of vulnerability detection techniques \textendash~ systematic manual penetration testing (SMPT), exploratory manual penetration testing (EMPT), dynamic application security testing (DAST), and static application security testing (SAST) \textendash\ to an open-source medical records system. RESULTS: We found the most vulnerabilities using SAST. However, EMPT found more severe vulnerabilities. With each technique, we found unique vulnerabilities not found using the other techniques. The efficiency of manual techniques (EMPT, SMPT) was comparable to or better than the efficiency of automated techniques (DAST, SAST) in terms of Vulnerabilities per Hour (VpH). CONCLUSIONS: The vulnerability detection technique practitioners should select may vary based on the goals and available resources of the project. If the goal of an organization is to find "all" vulnerabilities in a project, they need to use as many techniques as their resources allow.