A Large Scale Study of AI-based Binary Function Similarity Detection Techniques for Security Researchers and Practitioners

TL;DR

This large-scale empirical study evaluates nine AI-based binary function similarity detection tools using high-quality datasets, revealing their limitations, challenges, and potential for combined performance improvements in security applications.

Contribution

First comprehensive large-scale evaluation of AI-based BFSD tools with new datasets, analyzing performance factors, limitations, and proposing a strategy to enhance detection accuracy.

Findings

Improved overall performance by 13.4% using combined tools

Identified key challenges and limitations of current BFSD methods

Provided high-quality datasets for future research

Abstract

Binary Function Similarity Detection (BFSD) is a foundational technique in software security, underpinning a wide range of applications including vulnerability detection, malware analysis. Recent advances in AI-based BFSD tools have led to significant performance improvements. However, existing evaluations of these tools suffer from three key limitations: a lack of in-depth analysis of performance-influencing factors, an absence of realistic application analysis, and reliance on small-scale or low-quality datasets. In this paper, we present the first large-scale empirical study of AI-based BFSD tools to address these gaps. We construct two high-quality and diverse datasets: BinAtlas, comprising 12,453 binaries and over 7 million functions for capability evaluation; and BinAres, containing 12,291 binaries and 54 real-world 1-day vulnerabilities for evaluating vulnerability detection performance in practical IoT firmware settings. Using these datasets, we evaluate nine representative BFSD tools, analyze the challenges and limitations of existing BFSD tools, and investigate the consistency among BFSD tools. We also propose an actionable strategy for combining BFSD tools to enhance overall performance (an improvement of 13.4%). Our study not only advances the practical adoption of BFSD tools but also provides valuable resources and insights to guide future research in scalable and automated binary similarity detection.