On Evaluating the Renaissance Benchmarking Suite: Variety, Performance, and Complexity

TL;DR

This paper evaluates the Renaissance benchmarking suite's variety, performance, and complexity, providing detailed experimental analysis to assess its suitability for compiler optimization research.

Contribution

It offers a comprehensive assessment of the Renaissance suite's diversity and complexity, and its effectiveness for evaluating JVM compiler optimizations.

Findings

Renaissance exhibits high variety and complexity.

Performance differences between JITs are more pronounced on Renaissance.

The suite is suitable for testing new compiler optimizations.

Abstract

The recently proposed Renaissance suite is composed of modern, real-world, concurrent, and object-oriented workloads that exercise various concurrency primitives of the JVM. Renaissance was used to compare performance of two stateof-the-art, production-quality JIT compilers (HotSpot C2 and Graal), and to show that the performance differences are more significant than on existing suites such as DaCapo and SPECjvm2008. In this technical report, we give an overview of the experimental setup that we used to assess the variety and complexity of the Renaissance suite, as well as its amenability to new compiler optimizations. We then present the obtained measurements in detail.