Game-Theoretic Malware Detection

TL;DR

This paper introduces a game-theoretic approach to optimize the random selection of malware detection tools, balancing coverage and cost while accounting for strategic attacker behavior.

Contribution

It models malware detection as a Stackelberg security game and computes optimal randomized strategies for defenders to improve detection coverage.

Findings

Outperforms baseline strategies in empirical tests

Effective in diverse attack scenarios

Balances detection coverage and resource constraints

Abstract

Malware attacks are costly. To mitigate against such attacks, organizations deploy malware detection tools that help them detect and eventually resolve those threats. While running only the best available tool does not provide enough coverage of the potential attacks, running all available tools is prohibitively expensive in terms of financial cost and computing resources. Therefore, an organization typically runs a set of tools that maximizes their coverage given a limited budget. However, how should an organization choose that set? Attackers are strategic, and will change their behavior to preferentially exploit the gaps left by a deterministic choice of tools. To avoid leaving such easily-exploited gaps, the defender must choose a random set. In this paper, we present an approach to compute an optimal randomization over size-bounded sets of available security analysis tools by modeling the relationship between attackers and security analysts as a leader-follower Stackelberg security game. We estimate the parameters of our model by combining the information from the VirusTotal dataset with the more detailed reports from the National Vulnerability Database. In an empirical comparison, our approach outperforms a set of natural baselines under a wide range of assumptions.