Mutation-based Evaluation of Cryptographic API Misuse Detectors

TL;DR

This paper introduces the MASC framework, a mutation testing-based approach for systematically evaluating the effectiveness of cryptographic API misuse detectors, revealing significant undocumented flaws across multiple tools.

Contribution

The paper extends the MASC framework with new mutation operators and scopes, providing a comprehensive, data-driven evaluation method for crypto-detectors that uncovers previously unknown flaws.

Findings

Discovered 6 new undocumented flaws in crypto-detectors

Evaluated 14 crypto-detectors, including 5 new ones since 2022

Flaws affect detectors across open-source, industry, and research origins

Abstract

The correct use of cryptography is central to ensuring data security in modern software systems. Hence, several academic and commercial static analysis tools have been developed for detecting and mitigating crypto-API misuse. While developers are optimistically adopting these crypto-API misuse detectors (or crypto-detectors) in their software development cycles, this momentum must be accompanied by a rigorous understanding of their effectiveness at finding crypto-API misuse in practice. This paper describes the MASC framework, which enables a systematic and data-driven evaluation of crypto-detectors using mutation testing. We ground MASC in a comprehensive view of the problem space by developing a data-driven taxonomy of existing crypto-API misuse, containing 107 misuse cases organized among nine semantic clusters. We develop 19 generalizable usage-based mutation operators and three mutation scopes that can expressively instantiate thousands of compilable variants of the misuse cases for thoroughly evaluating crypto-detectors. Using MASC, in a previous study, we evaluated nine major crypto-detectors and discovered 19 unique, undocumented flaws that severely impact the ability of crypto-detectors to discover misuses in practice. This paper substantially extends our MASC framework and offers updated evaluation of the crypto-detectors in our 2022 study, in addition to 5 more, major crypto-detectors. Through this work, we find 6 new, undocumented flaws, and demonstrate that these flaws affect the crypto-detectors regardless of their origin; open-source community, industry, and/or research. We conclude with a discussion on the diverse perspectives that influence the design of crypto-detectors and future directions towards building security-focused crypto-detectors by design.