SeqShield: A Behavioral Analysis Approach to Uncover Rootkits

TL;DR

SeqShield is a dynamic, behavior-based rootkit detection method for Windows that analyzes API call sequences with machine learning, achieving high accuracy even against obfuscated variants.

Contribution

It introduces a novel API sequence analysis approach combined with feature optimization and machine learning to detect metamorphic rootkits effectively.

Findings

Random Forest achieves 97.27% accuracy with bigram features.

Metamorphic code variants are effectively detected using SeqShield.

Feature importance ranking improves detection efficiency without losing accuracy.

Abstract

Rootkits are among the most elusive types of malware, capable of bypassing traditional static analysis methods due to their metamorphic behavior. Signature-based detection techniques struggle against these threats, necessitating a shift toward dynamic analysis approaches. We propose SeqShield, a behavior-based rootkit detection approach designed specifically for the Windows OS, leveraging API call sequences for dynamic behavior analysis. Instead of relying on static signatures, SeqShield examines the execution patterns of API calls, which inherently reflect malicious intent. Analyzing API sequences, we can effectively identify rootkit-like behavior. We also employed a metamorphic code engine to generate 10X mutated variants of rootkits, demonstrating their obfuscation strategies. SeqShield applies n-gram analysis to extract bigram and trigram features from these API call sequences, enabling effective detection of rootkit-like activity. Among the models tested, Random Forest achieves the highest accuracy of 97.27% (bigram) and 96.17% (trigram). To optimize performance and decrease the dimension, we apply feature importance ranking using the Gini Impurity Index, iteratively selecting the most significant features. The optimized lower-dimensional feature matrix significantly enhances detection efficiency without sacrificing accuracy. Using the optimized feature set, our approach achieves 96.72% accuracy for bigrams and 97.81% accuracy for trigrams.