PALM: A Efficient Performance Simulator for Tiled Accelerators with Large-scale Model Training

TL;DR

PALM is a performance simulator designed for tiled accelerators used in large-scale deep learning training, enabling efficient prediction of optimal configurations and supporting various parallelism strategies.

Contribution

The paper introduces PALM, a novel, configurable performance simulator for tiled accelerators that models complex interactions and supports multiple parallelism strategies.

Findings

Supports user-configurable parallelism types

Models on-chip and off-chip memory interactions

Predicts performance throughput accurately

Abstract

Deep learning (DL) models are piquing high interest and scaling at an unprecedented rate. To this end, a handful of tiled accelerators have been proposed to support such large-scale training tasks. However, these accelerators often incorporate numerous cores or tiles even extending to wafer-scale, substantial on-chip bandwidth, and distributed memory systems. This results in an exceedingly complex design space. Moreover, conducting actual training experiments to find optimal configurations is impractical due to time constraints. Hence, predicting the optimal mapping of various parallelisms to such tiled system architectures becomes crucial. In this study, leveraging an analysis of existing mainstream DL model training strategies, we introduce a performance simulator named PALM. PALM targets both the training and inference processes for tiled accelerators, aiming to inspire the design of current and future accelerators. Specifically, (i) we establish a scheduling mechanism among tiled accelerators based on an event-driven framework; (ii) we support user-configurable pipeline, tensor, and data parallelism on tiled accelerators, determining the absolute performance throughput under these parallelism strategies; (iii) we model the interaction of on-chip SRAM, NoC, and off-chip DRAM during operator execution. This work is available here: https://github.com/fangjh21/PALM.