RX-INT: A Kernel Engine for Real-Time Detection and Analysis of In-Memory Threats

TL;DR

RX-INT is a kernel-based system that enhances real-time detection of in-memory threats like fileless malware by combining thread monitoring, memory hashing, and heuristics, outperforming existing tools such as PE-sieve.

Contribution

Introduces RX-INT, a novel kernel-assisted architecture with a resilient detection engine for real-time identification of in-memory threats, addressing limitations of prior methods.

Findings

Higher detection rate than PE-sieve in benchmarks

Successfully detects manually mapped malicious regions

Provides resilience against TOCTOU attacks

Abstract

Malware and cheat developers use fileless execution techniques to evade traditional, signature-based security products. These methods include various types of manual mapping, module stomping, and threadless injection which work entirely within the address space of a legitimate process, presenting a challenge for detection due to ambiguity between what is legitimate and what isn't. Existing tools often have weaknesses, such as a dependency on Portable Executable (PE) structures or a vulnerability to time-of-check-to-time-of-use (TOCTOU) race conditions where an adversary cleans up before a periodic scan has the chance to occur. To address this gap, we present RX-INT, a kernel-assisted system featuring an architecture that provides resilience against TOCTOU attacks. RX-INT introduces a detection engine that combines a real-time thread creation monitor with a stateful Virtual Address Descriptor (VAD) scanner alongside various heuristics within. This engine snapshots both private and image-backed memory regions, using real-time memory hashing to detect illicit modifications like module stomping. Critically, we demonstrate a higher detection rate in certain benchmarks of this approach through a direct comparison with PE-sieve, a commonly used and powerful memory forensics tool. In our evaluation, RX-INT successfully detected a manually mapped region that was not identified by PE-sieve. We then conclude that our architecture represents a tangible difference in the detection of fileless threats, with direct applications in the fields of anti-cheat and memory security.