Static Detection of Core Structures in Tigress Virtualization-Based Obfuscation Using an LLVM Pass

TL;DR

This paper presents a static analysis method using an LLVM Pass to identify core components of virtualization-based obfuscation in malware, aiding deobfuscation efforts.

Contribution

It introduces a novel LLVM-based static analysis technique to detect key virtualization structures in obfuscated code, improving analysis accuracy.

Findings

Successfully detects core virtualization structures across multiple modes

Effective in absence of compiler optimizations

Enhances deobfuscation capabilities

Abstract

Malware often uses obfuscation to hinder security analysis. Among these techniques, virtualization-based obfuscation is particularly strong because it protects programs by translating original instructions into attacker-defined virtual machine (VM) bytecode, producing long and complex code that is difficult to analyze and deobfuscate. This paper aims to identify the structural components of virtualization-based obfuscation through static analysis. By examining the execution model of obfuscated code, we define and detect the key elements required for deobfuscation-namely the dispatch routine, handler blocks, and the VM region-using LLVM IR. Experimental results show that, in the absence of compiler optimizations, the proposed LLVM Pass successfully detects all core structures across major virtualization options, including switch, direct, and indirect modes.