Finding Missed Code Size Optimizations in Compilers using LLMs

TL;DR

This paper presents a simple, extensible method using large language models and differential testing to identify missed code size optimizations in C/C++ compilers, successfully finding 24 bugs across multiple languages.

Contribution

It introduces a novel, minimalistic approach combining LLMs with differential testing to detect compiler optimization bugs, adaptable to multiple programming languages.

Findings

Reported 24 confirmed bugs in production compilers

Successfully extended approach from C/C++ to Rust and Swift

Approach requires fewer than 150 lines of code

Abstract

Compilers are complex, and significant effort has been expended on testing them. Techniques such as random program generation and differential testing have proved highly effective and have uncovered thousands of bugs in production compilers. The majority of effort has been expended on validating that a compiler produces correct code for a given input, while less attention has been paid to ensuring that the compiler produces performant code. In this work we adapt differential testing to the task of identifying missed optimization opportunities in compilers. We develop a novel testing approach which combines large language models (LLMs) with a series of differential testing strategies and use them to find missing code size optimizations in C / C++ compilers. The advantage of our approach is its simplicity. We offload the complex task of generating random code to an off-the-shelf LLM, and use heuristics and analyses to identify anomalous compiler behavior. Our approach requires fewer than 150 lines of code to implement. This simplicity makes it extensible. By simply changing the target compiler and initial LLM prompt we port the approach from C / C++ to Rust and Swift, finding bugs in both. To date we have reported 24 confirmed bugs in production compilers, and conclude that LLM-assisted testing is a promising avenue for detecting optimization bugs in real world compilers.