BenchPress: A Deep Active Benchmark Generator

TL;DR

BenchPress is a novel ML benchmark generator for compilers that actively steers source code synthesis towards desired features, outperforming existing methods in compilation success, diversity, and downstream task improvement.

Contribution

It introduces the first steerable, active learning-based benchmark generator for compiler source code, enabling targeted feature synthesis and improved downstream task performance.

Findings

Achieves 86% compilation rate, outperforming CLgen's 2.33%.

Generates 10x more unique, feature-diverse benchmarks.

Improves downstream CPU vs GPU heuristic model speedup.

Abstract

We develop BenchPress, the first ML benchmark generator for compilers that is steerable within feature space representations of source code. BenchPress synthesizes compiling functions by adding new code in any part of an empty or existing sequence by jointly observing its left and right context, achieving excellent compilation rate. BenchPress steers benchmark generation towards desired target features that has been impossible for state of the art synthesizers (or indeed humans) to reach. It performs better in targeting the features of Rodinia benchmarks in 3 different feature spaces compared with (a) CLgen - a state of the art ML synthesizer, (b) CLSmith fuzzer, (c) SRCIROR mutator or even (d) human-written code from GitHub. BenchPress is the first generator to search the feature space with active learning in order to generate benchmarks that will improve a downstream task. We show how using BenchPress, Grewe's et al. CPU vs GPU heuristic model can obtain a higher speedup when trained on BenchPress's benchmarks compared to other techniques. BenchPress is a powerful code generator: Its generated samples compile at a rate of 86%, compared to CLgen's 2.33%. Starting from an empty fixed input, BenchPress produces 10x more unique, compiling OpenCL benchmarks than CLgen, which are significantly larger and more feature diverse.