MLIR-Smith: A Novel Random Program Generator for Evaluating Compiler Pipelines

TL;DR

MLIR-Smith is a new random program generator designed to test and evaluate MLIR-based compiler pipelines, helping identify bugs and improve compiler robustness through differential testing.

Contribution

It introduces MLIR-Smith, the first tool capable of generating random MLIR programs for comprehensive compiler testing and evaluation.

Findings

Discovered multiple bugs in MLIR, LLVM, DaCe, and DCIR pipelines.

Demonstrated the utility of MLIR-Smith in differential testing.

Enhanced understanding of compiler robustness and testing needs.

Abstract

Compilers are essential for the performance and correct execution of software and hold universal relevance across various scientific disciplines. Despite this, there is a notable lack of tools for testing and evaluating them, especially within the adaptable Multi-Level Intermediate Representation (MLIR) context. This paper addresses the need for a tool that can accommodate MLIR's extensibility, a feature not provided by previous methods such as Csmith. Here we introduce MLIR-Smith, a novel random program generator specifically designed to test and evaluate MLIR-based compiler optimizations. We demonstrate the utility of MLIR-Smith by conducting differential testing on MLIR, LLVM, DaCe, and DCIR, which led to the discovery of multiple bugs in these compiler pipelines. The introduction of MLIR-Smith not only fills a void in the realm of compiler testing but also emphasizes the importance of comprehensive testing within these systems. By providing a tool that can generate random MLIR programs, this paper enhances our ability to evaluate and improve compilers and paves the way for future tools, potentially shaping the wider landscape of software testing and quality assurance.