Secure Development of a Hooking-Based Deception Framework Against Keylogging Techniques

TL;DR

This paper introduces a deception framework using API hooking to inject decoy keystrokes against keyloggers, including advanced anti-hooking techniques, demonstrating effectiveness and stealth in resisting real-world malware.

Contribution

It presents a resilient hooking-based deception system that counters anti-hooking strategies, ensuring continuous operation and deception against sophisticated keyloggers.

Findings

Successfully resists advanced anti-hooking keyloggers

Maintains stealth and operational continuity

Effective against real-world malware samples

Abstract

Keyloggers remain a serious threat in modern cybersecurity, silently capturing user keystrokes to steal credentials and sensitive information. Traditional defenses focus mainly on detection and removal, which can halt malicious activity but do little to engage or mislead adversaries. In this paper, we present a deception framework that leverages API hooking to intercept input-related API calls invoked by keyloggers at runtime and inject realistic decoy keystrokes. A core challenge, however, lies in the increasing adoption of anti-hooking techniques by advanced keyloggers. Anti-hooking strategies allow malware to bypass or detect instrumentation. To counter this, we introduce a hardened hooking layer that detects tampering and rapidly reinstates disrupted hooks, ensuring continuity of deception. We evaluate our framework against a custom-built "super keylogger" incorporating multiple evasion strategies, as well as 50 real-world malware samples spanning ten prominent keylogger families. Experimental results demonstrate that our system successfully resists sophisticated bypass attempts, maintains operational stealth, and reliably deceives attackers by feeding them decoys. The system operates with negligible performance overhead and no observable impact on user experience. Our findings show that resilient, runtime deception can play a practical and robust role in confronting advanced threats.