Adaptive Detection of Polymorphic Malware: Leveraging Mutation Engines and YARA Rules for Enhanced Security

TL;DR

This paper presents a framework for analyzing polymorphic malware detection techniques across multiple layers, demonstrating that hybrid detection approaches significantly improve detection rates over traditional methods.

Contribution

It introduces a reproducible framework for evaluating polymorphic malware detection across different layers and shows the effectiveness of hybrid detection strategies.

Findings

AVs detect 34% of polymorphic variants on average

YARA/Sigma and EDR detect around 74-76%

Integrated detection achieves approximately 92% detection with low false positives

Abstract

Polymorphic malware continually alters its structure to evade signature-based defences, challenging both commercial antivirus (AV) and enterprise detection systems. This study introduces a reproducible framework for analysing eight polymorphic behaviours-junk code insertion, control-flow obfuscation, packing, data encoding, domain generation, randomized beacon timing, protocol mimicry, and format/header tweaks-and evaluates their detectability across three layers: commercial AVs, custom rule-based detectors (YARA/Sigma), and endpoint detection and response (EDR) telemetry. Eleven inert polymorphic variants were generated per behaviour using controlled mutation engines and executed in isolated environments. Detection performance was assessed by detection rate (DR), false positive rate (FPR), and combined coverage. AVs achieved an average DR of 34%, YARA/Sigma 74% and EDR 76%; integrated detection reached ~92% with an FPR of 3.5%. Iterative YARA tuning showed a trade-off between detection and FPR, while behaviour-specific trends revealed static polymorphisms were best caught by custom rules, dynamic by EDR, and network-level by Sigma-like analysis. These results affirm that hybrid detection pipelines combining static, dynamic, and network-layer analytics offer resilient defence against polymorphic malware and form a baseline for future adaptive detection research.