Bootstrapping Fuzzers for Compilers of Low-Resource Language Dialects Using Language Models

TL;DR

This paper introduces Germinator, a novel approach that uses language models and grammar-based fuzzing to automatically generate effective test inputs for compiler dialects, improving bug detection and coverage in low-resource dialects.

Contribution

The paper presents Germinator, a tool that combines grammar extraction and language models to generate diverse seed inputs for dialect-agnostic fuzzing of compilers, addressing limitations of prior methods.

Findings

Seeds improve line coverage by 10-120% over baselines.

Discovered 88 previously unknown bugs, including 23 in low-resource dialects.

Effective testing of heterogeneous dialect ecosystems at scale.

Abstract

Modern extensible compiler frameworks-such as MLIR-enable rapid creation of domain-specific language dialects. This flexibility, however, makes correctness harder to ensure as the same extensibility that accelerates development also complicates maintaining the testing infrastructure. Extensible languages require automated test generation that is both dialect-agnostic (works across dialects without manual adaptation) and dialect-effective (targets dialect-specific features to find bugs). Existing approaches typically sacrifice one of these goals by either requiring manually constructed seed corpora for each dialect, or by failing to be effective. We present a dialect-agnostic and dialect-effective grammar-based and coverage-guided fuzzing approach for extensible compilers that combines two key insights from existing work: (i) the grammars of dialects, which already encode the structural and type constraints, can often be extracted automatically from the dialect specification; and (ii) these grammars can be used in combination with pre-trained large language models to automatically generate representative and diverse seed inputs from the full dialect space without requiring any manual input or training data. These seeds can then be used to bootstrap coverage-guided fuzzers. We built this approach into a tool, Germinator. When evaluated on six MLIR projects spanning 91 dialects, Germinator generated seeds improve line coverage by 10-120% over grammar-based baselines. We compare against grammar-based baselines because they are the only class of existing automatic seed generators that can be applied uniformly across MLIR's heterogeneous dialect ecosystem. Germinator discovers 88 previously unknown bugs (40 confirmed), including 23 in dialects with no prior automated test generators, demonstrating effective and controllable testing of low-resource dialects at scale.