Packed Malware Detection Using Grayscale Binary-to-Image Representations

TL;DR

This paper compares classical feature-based and deep learning methods for detecting packed malware using grayscale binary-to-image representations, demonstrating that CNNs significantly improve detection accuracy and robustness over traditional techniques.

Contribution

It introduces a novel approach combining grayscale byte-plot images with CNNs for effective packed malware detection, outperforming classical feature-based models.

Findings

CNN models outperform classical methods in detection accuracy.

DenseNet121 achieves higher precision and lower false positives.

The approach generalizes well to unknown packers.

Abstract

Detecting packed executables is a critical step in malware analysis, as packing obscures the original code and complicates static inspection. This study evaluates both classical feature-based methods and deep learning approaches that transform binary executables into visual representations, specifically, grayscale byte plots, and employ convolutional neural networks (CNNs) for automated classification of packed and non-packed binaries. A diverse dataset of benign and malicious Portable Executable (PE) files, packed using various commercial and open-source packers, was curated to capture a broad spectrum of packing transformations and obfuscation techniques. Classical models using handcrafted Gabor jet features achieved intense discrimination at moderate computational cost. In contrast, CNNs based on VGG16 and DenseNet121 significantly outperformed them, achieving high detection performance with well-balanced precision, recall, and F1-scores. DenseNet121 demonstrated slightly higher precision and lower false positive rates, whereas VGG16 achieved marginally higher recall, indicating complementary strengths for practical deployment. Evaluation against unknown packers confirmed robust generalization, demonstrating that grayscale byte-plot representations combined with deep learning provide a useful and reliable approach for early detection of packed malware, enhancing malware analysis pipelines and supporting automated antivirus inspection.