On the Effectiveness of Adversarial Samples against Ensemble Learning-based Windows PE Malware Detectors

TL;DR

This paper introduces FeaGAN, a novel adversarial attack method combining GANs and reinforcement learning to evade ensemble learning-based Windows PE malware detectors, achieving high success in preserving malware functionality.

Contribution

The study develops FeaGAN, an advanced mutation system that overcomes MalGAN limitations by integrating RL, enhancing the ability to evade ensemble malware detectors while maintaining malware integrity.

Findings

100% format preservation in generated malware samples

Achieved stable success in executability and maliciousness preservation

Demonstrated effectiveness against ensemble learning-based detectors

Abstract

Recently, there has been a growing focus and interest in applying machine learning (ML) to the field of cybersecurity, particularly in malware detection and prevention. Several research works on malware analysis have been proposed, offering promising results for both academic and practical applications. In these works, the use of Generative Adversarial Networks (GANs) or Reinforcement Learning (RL) can aid malware creators in crafting metamorphic malware that evades antivirus software. In this study, we propose a mutation system to counteract ensemble learning-based detectors by combining GANs and an RL model, overcoming the limitations of the MalGAN model. Our proposed FeaGAN model is built based on MalGAN by incorporating an RL model called the Deep Q-network anti-malware Engines Attacking Framework (DQEAF). The RL model addresses three key challenges in performing adversarial attacks on Windows Portable Executable malware, including format preservation, executability preservation, and maliciousness preservation. In the FeaGAN model, ensemble learning is utilized to enhance the malware detector's evasion ability, with the generated adversarial patterns. The experimental results demonstrate that 100\% of the selected mutant samples preserve the format of executable files, while certain successes in both executability preservation and maliciousness preservation are achieved, reaching a stable success rate.