Is Function Similarity Over-Engineered? Building a Benchmark

TL;DR

This paper introduces REFuSE-Bench, a new benchmark for binary function similarity detection that reveals simple byte-based methods outperform complex models, challenging current assumptions in the field.

Contribution

The paper presents a new benchmark dataset for binary function similarity, addressing data quality issues and evaluating ML models on Windows data, highlighting the effectiveness of simple byte-based approaches.

Findings

Byte-based models achieve state-of-the-art performance.

Complex models do not significantly outperform simple byte comparisons.

The benchmark reflects real-world malware analysis scenarios.

Abstract

Binary analysis is a core component of many critical security tasks, including reverse engineering, malware analysis, and vulnerability detection. Manual analysis is often time-consuming, but identifying commonly-used or previously-seen functions can reduce the time it takes to understand a new file. However, given the complexity of assembly, and the NP-hard nature of determining function equivalence, this task is extremely difficult. Common approaches often use sophisticated disassembly and decompilation tools, graph analysis, and other expensive pre-processing steps to perform function similarity searches over some corpus. In this work, we identify a number of discrepancies between the current research environment and the underlying application need. To remedy this, we build a new benchmark, REFuSE-Bench, for binary function similarity detection consisting of high-quality datasets and tests that better reflect real-world use cases. In doing so, we address issues like data duplication and accurate labeling, experiment with real malware, and perform the first serious evaluation of ML binary function similarity models on Windows data. Our benchmark reveals that a new, simple basline, one which looks at only the raw bytes of a function, and requires no disassembly or other pre-processing, is able to achieve state-of-the-art performance in multiple settings. Our findings challenge conventional assumptions that complex models with highly-engineered features are being used to their full potential, and demonstrate that simpler approaches can provide significant value.