User-space library rootkits revisited: Are user-space detection mechanisms futile?

TL;DR

This paper challenges the effectiveness of user-space detection tools for user-space rootkits, demonstrating through experiments that such rootkits can evade detection and emphasizing the need for cautious communication of detection results.

Contribution

It provides experimental evidence that user-space rootkits can evade detection by standard tools and argues that detection in user-space is fundamentally futile, offering guidelines for improved detection strategies.

Findings

User-space rootkits can bypass popular detection tools.

Detection results should be communicated with caution.

Traditional detection methods are insufficient against sophisticated rootkits.

Abstract

The kind of malware designed to conceal malicious system resources (e.g. processes, network connections, files, etc.) is commonly referred to as a rootkit. This kind of malware represents a significant threat in contemporany systems. Despite the existence of kernel-space rootkits (i.e. rootkits that infect the operating system kernel), user-space rootkits (i.e. rootkits that infect the user-space operating system tools, commands and libraries) continue to pose a significant danger. However, kernel-space rootkits attract all the attention, implicitly assuming that user-space rootkits (malware that is still in existence) are easily detectable by well-known user-space tools that look for anomalies. The primary objective of this work is to answer the following question: Is detecting user-space rootkits with user-space tools futile? Contrary to the prevailing view that considers it effective, we argue that the detection of user-space rootkits cannot be done in user-space at all. Moreover, the detection results must be communicated to the user with extreme caution. To support this claim, we conducted different experiments focusing on process concealing in Linux systems. In these experiments, we evade the detection mechanisms widely accepted as the standard solution for this type of user-space malware, bypassing the most popular open source anti-rootkit tool for process hiding. This manuscript describes the classical approach to build user-space library rootkits, the traditional detection mechanisms, and different evasion techniques (it also includes understandable code snippets and examples). In addition, it offers some guidelines to implement new detection tools and improve the existing ones to the extent possible.