PerfGen: Automated Performance Benchmark Generation for Big Data Analytics

TL;DR

PerfGen is an automated tool that generates inputs to trigger performance issues in big data analytics, using phased fuzzing, guidance metrics, and skew-inspired mutations to efficiently identify problematic workloads.

Contribution

It introduces a novel phased fuzzing approach with guidance metrics and input transformations to effectively generate performance-triggering inputs in big data analytics.

Findings

Achieves at least 11x speedup over traditional fuzzing.

Generates workload inputs in less than 0.004% of baseline iterations.

Successfully identifies performance symptoms in four case studies.

Abstract

Many symptoms of poor performance in big data analytics such as computational skews, data skews, and memory skews are input dependent. However, due to the lack of inputs that can trigger such performance symptoms, it is hard to debug and test big data analytics. We design PerfGen to automatically generate inputs for the purpose of performance testing. PerfGen overcomes three challenges when naively using automated fuzz testing for the purpose of performance testing. First, typical greybox fuzzing relies on coverage as a guidance signal and thus is unlikely to trigger interesting performance behavior. Therefore, PerfGen provides performance monitor templates that a user can extend to serve as a set of guidance metrics for grey-box fuzzing. Second, performance symptoms may occur at an intermediate or later stage of a big data analytics pipeline. Thus, PerfGen uses a phased fuzzing approach. This approach identifies symptom-causing intermediate inputs at an intermediate stage first and then converts them to the inputs at the beginning of the program with a pseudo-inverse function generated by a large language model. Third, PerfGen defines sets of skew-inspired input mutations, which increases the chance of inducing performance problems. We evaluate PerfGen using four case studies. PerfGen achieves at least 11x speedup compared to a traditional fuzzing approach when generating inputs to trigger performance symptoms. Additionally, identifying intermediate inputs first and then converting them to original inputs enables PerfGen to generate such workloads in less than 0.004% of the iterations required by a baseline approach.