Virtual Breakpoints for x86/64

TL;DR

This paper introduces Virtual Breakpoints, a hardware-based mechanism integrated into the x86 MMU, to improve the reliability and efficiency of program execution trapping for dynamic malware analysis, overcoming limitations of traditional methods.

Contribution

It presents a novel hardware modification to the x86 MMU that manages breakpoints directly in hardware, enhancing speed and reliability over traditional binary modification techniques.

Findings

Hardware-based breakpoints are faster and more reliable.

Current trapping methods have fundamental abstraction failures.

The proposed design leverages 50 years of virtualization insights.

Abstract

Efficient, reliable trapping of execution in a program at the desired location is a linchpin technique for dynamic malware analysis. The progression of debuggers and malware is akin to a game of cat and mouse - each are constantly in a state of trying to thwart one another. At the core of most efficient debuggers today is a combination of virtual machines and traditional binary modification breakpoints (int3). In this paper, we present a design for Virtual Breakpoints. a modification to the x86 MMU which brings breakpoint management into hardware alongside page tables. In this paper we demonstrate the fundamental abstraction failures of current trapping methods, and design a new mechanism from the hardware up. This design incorporates lessons learned from 50 years of virtualization and debugger design to deliver fast, reliable trapping without the pitfalls of traditional binary modification.