A Deep Dive into Function Inlining and its Security Implications for ML-based Binary Analysis

TL;DR

This paper investigates how function inlining in compilers affects machine learning-based binary analysis security, revealing that aggressive inlining can alter static features and impact ML model robustness, with implications for binary analysis reliability.

Contribution

It provides the first comprehensive analysis of function inlining's security implications for ML-based binary analysis, including systematic evaluation under extreme inlining scenarios.

Findings

Inlining can affect ML model behaviors and security.

Static feature-based models are highly sensitive to inlining.

Compiler settings can be exploited to create evasive binaries.

Abstract

A function inlining optimization is a widely used transformation in modern compilers, which replaces a call site with the callee's body in need. While this transformation improves performance, it significantly impacts static features such as machine instructions and control flow graphs, which are crucial to binary analysis. Yet, despite its broad impact, the security impact of function inlining remains underexplored to date. In this paper, we present the first comprehensive study of function inlining through the lens of machine learning-based binary analysis. To this end, we dissect the inlining decision pipeline within the LLVM's cost model and explore the combinations of the compiler options that aggressively promote the function inlining ratio beyond standard optimization levels, which we term extreme inlining. We focus on five ML-assisted binary analysis tasks for security, using 20 unique models to systematically evaluate their robustness under extreme inlining scenarios. Our extensive experiments reveal several significant findings: i) function inlining, though a benign transformation in intent, can (in)directly affect ML model behaviors, being potentially exploited by evading discriminative or generative ML models; ii) ML models relying on static features can be highly sensitive to inlining; iii) subtle compiler settings can be leveraged to deliberately craft evasive binary variants; and iv) inlining ratios vary substantially across applications and build configurations, undermining assumptions of consistency in training and evaluation of ML models.