Validating Simplified Processor Models in Architectural Studies

TL;DR

This paper introduces a validation methodology for simplified processor models in architectural studies, focusing on trend consistency across benchmarks rather than absolute accuracy, demonstrated through a case study with FPGA-accelerated simulation.

Contribution

It proposes a new trend-based validation approach for simplified models and demonstrates its effectiveness through a comprehensive case study comparing different core models.

Findings

Simplified core models generally produce similar qualitative results as cycle-accurate models.

Most mismatches are within experimental noise or inconclusive policy differences.

Validation requires a detailed cycle-accurate model for reliable results.

Abstract

Cycle-accurate software simulation of multicores with complex microarchitectures is often excruciatingly slow. People use simplified core models to gain simulation speed. However, a persistent question is to what extent the results derived from a simplified core model can be used to characterize the behavior of a real machine. We propose a new methodology of validating simplified simulation models, which focuses on the trends of metric values across benchmarks and architectures, instead of errors of absolute metric values. To illustrate this methodology, we conduct a case study using an FPGA-accelerated cycle-accurate full system simulator. We evaluated three cache replacement polices on a 10-stage in-order core model, and then re-conducted all the experiments by substituting a 1-IPC core model for the 10-stage core model. We found that the 1-IPC core model generally produces qualitatively the same results as the accurate core model except for a few mismatches. We argue that most observed mismatches were either indistinguishable from experimental noise or corresponded to the cases where the policy differences even in the accurate model showed inconclusive results. We think it is fair to use simplified core models to study a feature once the influence of the simplification is understood. Additional studies on branch predictors and scaling properties of multithread benchmarks reinforce our argument. However, the validation of a simplified model requires a detailed cycle-accurate model!