MalMoE: Mixture-of-Experts Enhanced Encrypted Malicious Traffic Detection Under Graph Drift

TL;DR

MalMoE is a novel graph-assisted encrypted traffic detection system that uses Mixture of Experts to adaptively handle graph drift, improving real-time malicious traffic detection accuracy.

Contribution

Introduces MalMoE, a drift-aware detection system employing MoE and 1-hop-GNN experts with a two-stage training strategy for encrypted traffic analysis.

Findings

High detection accuracy on diverse datasets

Effective handling of graph drift in traffic analysis

Real-time detection capability

Abstract

Encryption has been commonly used in network traffic to secure transmission, but it also brings challenges for malicious traffic detection, due to the invisibility of the packet payload. Graph-based methods are emerging as promising solutions by leveraging multi-host interactions to promote detection accuracy. But most of them face a critical problem: Graph Drift, where the flow statistics or topological information of a graph change over time. To overcome these drawbacks, we propose a graph-assisted encrypted traffic detection system, MalMoE, which applies Mixture of Experts (MoE) to select the best expert model for drift-aware classification. Particularly, we design 1-hop-GNN-like expert models that handle different graph drifts by analyzing graphs with different features. Then, the redesigned gate model conducts expert selection according to the actual drift. MalMoE is trained with a stable two-stage training strategy with data augmentation, which effectively guides the gate on how to perform routing. Experiments on open-source, synthetic, and real-world datasets show that MalMoE can perform precise and real-time detection.