Understanding Simulated Architecture via gem5 Call-Stack Profiling

TL;DR

This paper introduces a lightweight call-stack profiling framework for gem5 that provides direct insights into simulated architecture behavior, revealing issues like cache deadlocks and inefficiencies.

Contribution

A novel, non-intrusive profiling method using Linux perf_event to analyze gem5's call-stacks, enabling detailed, hierarchical, and component-specific insights into simulation internals.

Findings

Uncovered inefficiencies in TimingSimpleCPU due to lockup-cache model

Detected cache coherence deadlock and livelock issues in gem5 simulations

Provided detailed call-tree views of gem5's internal activity

Abstract

Understanding the behavior of simulated architectures in gem5 is critical for studying complex, deeply integrated computing systems. However, conventional analysis methods provide only an indirect view of the simulated system internals. In this work, we show that call-stack profiling of gem5 itself offers a powerful yet underutilized perspective: the simulator's own call-stack directly reflects the activity of the simulated system, exposing insights that conventional statistics may overlook. Profiling gem5's call-stacks is challenging due to its highly layered and complex software design patterns. To address this, we introduce a specialized, lightweight profiling framework built on Linux's perf_event interface which samples gem5's runtime call-stacks throughout the simulation, resolves symbols on the fly, and merges samples into a hierarchical call-tree representation supporting both high-level structural views and focused, user-defined, component-specific analysis. Moreover, all profiling is performed in a separate process running alongside the main gem5 process, avoiding intrusive changes and overheads to the simulation itself. We apply our framework to gem5's three major CPU models -- AtomicSimpleCPU, TimingSimpleCPU, and O3CPU -- together with the Ruby memory system, and uncover behaviors that are not easily observable in conventional gem5 statistics. Our case studies reveal, for example, that TimingSimpleCPU is inefficient due to its use of a lockup-cache model and, despite its conceptual simplicity, does not simulate faster than a full out-of-order core. In addition, our tool makes it straightforward to detect cache coherence protocol deadlock and livelock -- issues that are otherwise difficult to identify, since the simulation either appears to run normally or terminates abruptly, making it hard to pinpoint when these conditions occur.