Achieving Consistent and Comparable CPU Evaluation

TL;DR

This paper presents a new CPU evaluation methodology that ensures consistent and comparable results by controlling for variability caused by other system components, addressing flaws in existing benchmarks like SPEC CPU2017.

Contribution

The paper introduces a rigorous, controlled CPU evaluation methodology validated through theoretical analysis and experiments, improving upon existing approaches.

Findings

Our methodology achieves higher consistency in CPU evaluation results.

Existing benchmarks like SPEC CPU2017 show significant variability due to uncontrolled factors.

Controlled experiments demonstrate the superiority of our approach over traditional methods.

Abstract

The challenge of CPU evaluation lies in the fact that user-perceived performance metrics can only be measured on an independently running system consisting of the CPU and other indispensable components, and hence it is difficult to accurately attribute the deviations in the evaluation outcomes to the differences between the CPUs. Our experiments reveal that the industry-standard CPU benchmark, SPEC CPU2017, suffers from a significant flaw: for the identical CPU, undefined configurations of other indispensable components introduce uncontrolled variability in evaluation outcomes. We propose a rigorous CPU evaluation methodology. Through theoretical analysis and pioneering controlled experiments, we systematically compare our methodology against four established methodologies: the SPEC CPU 2017, two DOE variants, and one RCTs approach. The results show our methodology can achieve consistent and comparable evaluation outcomes, while others exhibit inherent limations.