A Comparative Study of Software Secrets Reporting by Secret Detection Tools
Setu Kumar Basak, Jamison Cox, Bradley Reaves, Laurie Williams

TL;DR
This study compares the effectiveness of various secret detection tools on open-source and proprietary platforms, highlighting their strengths and weaknesses to guide developers in selecting appropriate tools and improving secret detection accuracy.
Contribution
It provides the first comprehensive empirical evaluation of multiple secret detection tools, analyzing their precision, recall, and common failure modes.
Findings
Top tools by precision: GitHub Secret Scanner, Gitleaks, Commercial X.
Top tools by recall: Gitleaks, SpectralOps, TruffleHog.
False positives often due to regex and entropy issues; false negatives due to regex faults and rule gaps.
Abstract
Background: According to GitGuardian's monitoring of public GitHub repositories, secrets sprawl continued accelerating in 2022 by 67% compared to 2021, exposing over 10 million secrets (API keys and other credentials). Though many open-source and proprietary secret detection tools are available, these tools output many false positives, making it difficult for developers to take action and teams to choose one tool out of many. To our knowledge, the secret detection tools are not yet compared and evaluated. Aims: The goal of our study is to aid developers in choosing a secret detection tool to reduce the exposure of secrets through an empirical investigation of existing secret detection tools. Method: We present an evaluation of five open-source and four proprietary tools against a benchmark dataset. Results: The top three tools based on precision are: GitHub Secret Scanner (75%),…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsDigital and Cyber Forensics · Advanced Malware Detection Techniques · Information and Cyber Security
