Accelerating OpenPangu Inference on NPU via Speculative Decoding

TL;DR

This paper proposes a speculative decoding acceleration scheme for OpenPangu-7B to improve inference speed on NPU hardware, addressing memory bottlenecks and lack of native support for such algorithms.

Contribution

It introduces an end-to-end speculative inference acceleration method tailored for OpenPangu-7B on NPU hardware, overcoming existing hardware limitations.

Findings

Significant speedup in inference time on NPU hardware

Effective mitigation of memory wall bottleneck

Compatibility with mainstream speculative decoding algorithms

Abstract

To mitigate the Memory Wall bottleneck encountered by Large Language Models (LLMs) during inference on \textbf{NPU} hardware, and addressing the scarcity of native support for mainstream speculative decoding algorithms on domestic infrastructure, this study presents an end-to-end speculative inference acceleration scheme for OpenPangu-7B.