Who is Debugging the Debuggers? Exposing Debug Information Bugs in Optimized Binaries

TL;DR

This paper introduces Debug$^{2}$, a framework that detects bugs in debug information of optimized binaries by differential analysis, revealing multiple bugs in popular compiler and debugger toolchains, some of which have been fixed.

Contribution

The paper presents a novel differential analysis framework, Debug$^{2}$, for identifying debug information bugs in modern toolchains, improving post-deployment debugging reliability.

Findings

Discovered 23 bugs in LLVM toolchain

Found 8 bugs in GNU toolchain

Identified 3 bugs in Rust toolchain

Abstract

Despite the advancements in software testing, bugs still plague deployed software and result in crashes in production. When debugging issues -- sometimes caused by "heisenbugs" -- there is the need to interpret core dumps and reproduce the issue offline on the same binary deployed. This requires the entire toolchain (compiler, linker, debugger) to correctly generate and use debug information. Little attention has been devoted to checking that such information is correctly preserved by modern toolchains' optimization stages. This is particularly important as managing debug information in optimized production binaries is non-trivial, often leading to toolchain bugs that may hinder post-deployment debugging efforts. In this paper, we present Debug$^{2}$, a framework to find debug information bugs in modern toolchains. Our framework feeds random source programs to the target toolchain and surgically compares the debugging behavior of their optimized/unoptimized binary variants. Such differential analysis allows Debug$^{2}$ to check invariants at each debugging step and detect bugs from invariant violations. Our invariants are based on the (in)consistency of common debug entities, such as source lines, stack frames, and function arguments. We show that, while simple, this strategy yields powerful cross-toolchain and cross-language invariants, which can pinpoint several bugs in modern toolchains. We have used Debug$^{2}$ to find 23 bugs in the LLVM toolchain (clang/lldb), 8 bugs in the GNU toolchain (GCC/gdb), and 3 in the Rust toolchain (rustc/lldb) -- with 14 bugs already fixed by the developers.