Multi-Language Benchmark Generation via L-Systems

TL;DR

This paper introduces a novel method using L-systems to generate multi-language benchmarks for testing compiler and system performance, demonstrated through various case studies comparing different languages and compiler optimizations.

Contribution

It presents a new approach to create artificial, complex program benchmarks across multiple programming languages using L-systems, enhancing performance evaluation tools.

Findings

Effective in generating complex multi-language benchmarks

Useful for comparing compiler performance and optimizations

Provides insights into code evolution and behavior

Abstract

L-systems are a mathematical formalism proposed by biologist Aristid Lindenmayer with the aim of simulating organic structures such as trees, snowflakes, flowers, and other branching phenomena. They are implemented as a formal language that defines how patterns can be iteratively rewritten. This paper describes how such a formalism can be used to create artificial programs written in programming languages such as C, C++, Julia and Go. These programs, being large and complex, can be used to test the performance of compilers, operating systems, and computer architectures. This paper demonstrates the usefulness of these benchmarks through multiple case studies. These case studies include a comparison between clang and gcc; a comparison between C, C++, Julia and Go; a study of the historical evolution of gcc in terms of code quality; a look into the effects of profile guided optimizations in gcc; an analysis of the asymptotic behavior of the different phases of clang's compilation pipeline; and a comparison between the many data structures available in the Gnome Library (GLib). These case studies demonstrate the benefits of the L-System approach to create benchmarks, when compared with fuzzers such as CSmith, which were designed to uncover bugs in compilers, rather than evaluating their performance.