Practical Testing of a C99 Compiler Using Output Comparison

TL;DR

This paper presents a practical output comparison technique for testing C99 compilers, enabling efficient bug detection and standards compliance verification across multiple compiler versions and sources.

Contribution

It introduces a simple, scalable method for compiler testing using output comparison, reducing manual effort and uncovering numerous bugs in various compilers.

Findings

Detected several hundred bugs in in-house and third-party code

Identified 18 bugs in GNU C compiler and specific bugs in GCC variants

Uncovered critical bugs in ARM and other platform-specific GCC versions

Abstract

A simple technique is presented for testing a C99 compiler, by comparison of its output with output from preexisting tools. The advantage to this approach is that new test cases can be added in bulk from existing sources, reducing the need for in-depth investigation of correctness issues, and for creating new test code by hand. This technique was used in testing the PalmSource Palm OS Cobalt ARM C/C++ cross-compiler for Palm-Powered personal digital assistants, primarily for standards-compliance and correct execution of generated code. The technique described here found several hundred bugs, mostly in our in-house code, but also in longstanding high-quality front- and back-end code from Edison Design Group and Apogee Software. It also found eighteen bugs in the GNU C compiler, as well as a bug specific to the Apple version of GCC, a bug specific to the Suse version of GCC, and a dozen bugs in versions of GCC for the ARM processor, several of them critical.