Analytic Roofline Modeling and Energy Analysis of LULESH Proxy Application on Multi-Core Clusters

TL;DR

This paper performs a detailed performance and energy analysis of the LULESH proxy application on multi-core clusters, using Roofline modeling and hardware counters to understand hot spot behaviors and energy efficiency.

Contribution

It introduces a comprehensive Roofline-based performance and energy model for LULESH on modern CPUs, comparing OpenMP and MPI implementations, and analyzing energy optimization strategies.

Findings

Memory-bound functions benefit from fewer cores and lower clock speeds for energy savings.

High baseline power limits the effectiveness of concurrency throttling.

Memory-bound workloads have lower EDP on Sapphire Rapids compared to Ice Lake.

Abstract

We present a thorough performance and energy consumption analysis of the LULESH proxy application in its OpenMP and MPI variants on two different clusters based on Intel Ice Lake (ICL) and Sapphire Rapids (SPR) CPUs. We first study the strong scaling and power consumption characteristics of the six hot spot functions in the code on the node level, with a special focus on memory bandwidth utilization. We then proceed with the construction of a detailed Roofline performance model for each memory-bound hot spot, which we validate using hardware performance counter measurements. We also comment on the observed discrepancies between the analytical model and the observations. To discern the influence of the programming model from the influence of implementation of the code, we compare the performance of OpenMP and MPI based on problem size, examining if the underlying implementation is equivalent for large problems, and if differences in overheads are more significant at smaller problem sizes. We also conduct an analysis of the power dissipation, energy to solution, and energy-delay product (EDP) of the hot spots, quantifying the influence of problem size, core and uncore clock frequency, and number of active cores per ccNUMA domain. Relevant energy savings are only possible for memory-bound functions by using fewer cores per ccNUMA domain and/or reducing the core clock speed. A major issue is the very high extrapolated baseline power on both chips, which makes concurrency throttling less effective. In terms of energy-delay product (EDP), on SPR only memory-bound workloads offer lower EDP compared to Ice Lake.