CBSeq: A Channel-level Behavior Sequence For Encrypted Malware Traffic Detection

TL;DR

CBSeq introduces a novel traffic representation and a Transformer-based classifier to improve encrypted malware traffic detection, especially for unknown variants, achieving higher accuracy and lower false positives.

Contribution

This paper presents CBSeq, a new method combining behavior sequence construction and MSFormer for enhanced malware traffic detection, including unknown variants.

Findings

Effective detection of known malware traffic.

Superior performance in unknown malware traffic detection.

Outperforms state-of-the-art methods.

Abstract

Machine learning and neural networks have become increasingly popular solutions for encrypted malware traffic detection. They mine and learn complex traffic patterns, enabling detection by fitting boundaries between malware traffic and benign traffic. Compared with signature-based methods, they have higher scalability and flexibility. However, affected by the frequent variants and updates of malware, current methods suffer from a high false positive rate and do not work well for unknown malware traffic detection. It remains a critical task to achieve effective malware traffic detection. In this paper, we introduce CBSeq to address the above problems. CBSeq is a method that constructs a stable traffic representation, behavior sequence, to characterize attacking intent and achieve malware traffic detection. We novelly propose the channels with similar behavior as the detection object and extract side-channel content to construct behavior sequence. Unlike benign activities, the behavior sequences of malware and its variant's traffic exhibit solid internal correlations. Moreover, we design the MSFormer, a powerful Transformer-based multi-sequence fusion classifier. It captures the internal similarity of behavior sequence, thereby distinguishing malware traffic from benign traffic. Our evaluations demonstrate that CBSeq performs effectively in various known malware traffic detection and exhibits superior performance in unknown malware traffic detection, outperforming state-of-the-art methods.