Optimization of Armv9 architecture general large language model inference performance based on Llama.cpp

TL;DR

This paper enhances the inference speed and efficiency of the Qwen-1.8B large language model on Armv9 architecture by applying quantization, operator vectorization, and compilation optimizations, achieving significant performance gains with minimal accuracy loss.

Contribution

It introduces specific optimization techniques for Llama.cpp on Armv9, including Int8 quantization and operator vectorization, to improve large language model inference performance.

Findings

Prefill performance increased by 1.6 times

Decoding performance increased by 24 times

Memory usage reduced to 1/5 of original

Abstract

This article optimizes the inference performance of the Qwen-1.8B model by performing Int8 quantization, vectorizing some operators in llama.cpp, and modifying the compilation script to improve the compiler optimization level. On the Yitian 710 experimental platform, the prefill performance is increased by 1.6 times, the decoding performance is increased by 24 times, the memory usage is reduced to 1/5 of the original, and the accuracy loss is almost negligible.