Evaluation of Static Vulnerability Detection Tools with Java Cryptographic API Benchmarks

TL;DR

This paper develops comprehensive benchmarks to evaluate the accuracy and scalability of static vulnerability detection tools for Java cryptographic API misuse, providing a basis for improving security solutions.

Contribution

It introduces two detailed benchmarks, CryptoAPI-Bench and ApacheCryptoAPI-Bench, for systematically assessing cryptographic misuse detection tools in Java.

Findings

CryptoGuard and CrySL outperform others in accuracy.

Benchmarks reveal scalability limits of current tools.

False-positive rates are effectively tested with correct cases.

Abstract

Several studies showed that misuses of cryptographic APIs are common in real-world code (e.g., Apache projects and Android apps). There exist several open-sourced and commercial security tools that automatically screen Java programs to detect misuses. To compare their accuracy and security guarantees, we develop two comprehensive benchmarks named CryptoAPI-Bench and ApacheCryptoAPI-Bench. CryptoAPI-Bench consists of 181 unit test cases that cover basic cases, as well as complex cases, including interprocedural, field sensitive, multiple class test cases, and path sensitive data flow of misuse cases. The benchmark also includes correct cases for testing false-positive rates. The ApacheCryptoAPI-Bench consists of 121 cryptographic cases from 10 Apache projects. We evaluate four tools, namely, SpotBugs, CryptoGuard, CrySL, and Coverity using both benchmarks. We present their performance and comparative analysis. The ApacheCryptoAPI-Bench also examines the scalability of the tools. Our benchmarks are useful for advancing state-of-the-art solutions in the space of misuse detection.