Assessing Tenstorrent's RISC-V MatMul Acceleration Capabilities

TL;DR

This paper evaluates the Tenstorrent Grayskull RISC-V accelerator's performance for matrix multiplication in AI workloads, highlighting its efficiency and trade-offs compared to leading GPU architectures.

Contribution

It provides a detailed characterization of Grayskull's execution model and compares its performance and power efficiency against top-tier GPU and CPU architectures.

Findings

Grayskull achieves 1.55 TFLOPs/Watt with BF16 precision.

NVIDIA GPUs outperform in raw performance but consume more power.

Grayskull offers a competitive balance of power efficiency and throughput.

Abstract

The increasing demand for generative AI as Large Language Models (LLMs) services has driven the need for specialized hardware architectures that optimize computational efficiency and energy consumption. This paper evaluates the performance of the Tenstorrent Grayskull e75 RISC-V accelerator for basic linear algebra kernels at reduced numerical precision, a fundamental operation in LLM computations. We present a detailed characterization of Grayskull's execution model, gridsize, matrix dimensions, data formats, and numerical precision impact computational efficiency. Furthermore, we compare Grayskull's performance against state-of-the-art architectures with tensor acceleration, including Intel Sapphire Rapids processors and two NVIDIA GPUs (V100 and A100). Whilst NVIDIA GPUs dominate raw performance, Grayskull demonstrates a competitive trade-off between power consumption and computational throughput, reaching a peak of 1.55 TFLOPs/Watt with BF16.