ONNXim: A Fast, Cycle-level Multi-core NPU Simulator

TL;DR

ONNXim is a high-speed, cycle-level multi-core NPU simulator that efficiently models DRAM, NoC, and multi-tenancy, enabling practical research and design space exploration for DNN serving systems.

Contribution

It introduces a fast, cycle-level multi-core NPU simulator that supports ONNX models and multi-tenancy, overcoming limitations of existing tools.

Findings

Significantly faster than existing simulators (up to 384x)

Supports detailed DRAM and NoC modeling for contention analysis

Enables practical multi-tenant NPU case studies

Abstract

As DNNs are widely adopted in various application domains while demanding increasingly higher compute and memory requirements, designing efficient and performant NPUs (Neural Processing Units) is becoming more important. However, existing architectural NPU simulators lack support for high-speed simulation, multi-core modeling, multi-tenant scenarios, detailed DRAM/NoC modeling, and/or different deep learning frameworks. To address these limitations, this work proposes ONNXim, a fast cycle-level simulator for multi-core NPUs in DNN serving systems. It takes DNN models represented in the ONNX graph format generated from various deep learning frameworks for ease of simulation. In addition, based on the observation that typical NPU cores process tensor tiles from on-chip scratchpad memory with deterministic compute latency, we forgo a detailed modeling for the computation while still preserving simulation accuracy. ONNXim also preserves dependencies between compute and tile DMAs. Meanwhile, the DRAM and NoC are modeled in cycle-level to properly model contention among multiple cores that can execute different DNN models for multi-tenancy. Consequently, ONNXim is significantly faster than existing simulators (e.g., by up to 384x over Accel-sim) and enables various case studies, such as multi-tenant NPUs, that were previously impractical due to slow speed and/or lack of functionalities. ONNXim is publicly available at https://github.com/PSAL-POSTECH/ONNXim.