Targeting Infeasibility Questions on Obfuscated Codes

TL;DR

This paper introduces Backward-Bounded DSE, a novel method for accurately solving infeasibility questions in code obfuscation, enhancing malware analysis and deobfuscation techniques.

Contribution

It presents a new backward-bounded dynamic symbolic execution approach that effectively addresses infeasibility questions, complementing existing DSE methods for better disassembly of obfuscated binaries.

Findings

Successfully applied to state-of-the-art packers and malware

Effectively detects opaque predicates and call stack tampering

Enables complete deobfuscation of complex malware like X-Tunnel

Abstract

Software deobfuscation is a crucial activity in security analysis and especially, in malware analysis. While standard static and dynamic approaches suffer from well-known shortcomings, Dynamic Symbolic Execution (DSE) has recently been proposed has an interesting alternative, more robust than static analysis and more complete than dynamic analysis. Yet, DSE addresses certain kinds of questions encountered by a reverser namely feasibility questions. Many issues arising during reverse, e.g. detecting protection schemes such as opaque predicates fall into the category of infeasibility questions. In this article, we present the Backward-Bounded DSE, a generic, precise, efficient and robust method for solving infeasibility questions. We demonstrate the benefit of the method for opaque predicates and call stack tampering, and give some insight for its usage for some other protection schemes. Especially, the technique has successfully been used on state-of-the-art packers as well as on the government-grade X-Tunnel malware -- allowing its entire deobfuscation. Backward-Bounded DSE does not supersede existing DSE approaches, but rather complements them by addressing infeasibility questions in a scalable and precise manner. Following this line, we propose sparse disassembly, a combination of Backward-Bounded DSE and static disassembly able to enlarge dynamic disassembly in a guaranteed way, hence getting the best of dynamic and static disassembly. This work paves the way for robust, efficient and precise disassembly tools for heavily-obfuscated binaries.