ArcLight: A Lightweight LLM Inference Architecture for Many-Core CPUs

TL;DR

ArcLight is a lightweight architecture optimized for many-core CPUs that improves large language model inference throughput by addressing cross-NUMA memory access and integrating efficient memory and thread management.

Contribution

It introduces a novel inference architecture that effectively exploits many-core CPU architectures, surpassing existing frameworks in performance and maintaining broad device compatibility.

Findings

Achieves up to 46% higher inference throughput than mainstream frameworks.

Effectively mitigates cross-NUMA memory access overhead.

Maintains compatibility with arbitrary CPU devices.

Abstract

Although existing frameworks for large language model (LLM) inference on CPUs are mature, they fail to fully exploit the computation potential of many-core CPU platforms. Many-core CPUs are widely deployed in web servers and high-end networking devices, and are typically organized into multiple NUMA nodes that group cores and memory. Current frameworks largely overlook the substantial overhead of cross-NUMA memory access, limiting inference scalability and intelligence enabling on such platforms. To address this limitation, we build ArcLight, a lightweight LLM inference architecture designed from the ground up for many-core CPUs. ArcLight integrates efficient memory management and thread scheduling, and introduces finely controlled tensor parallelism to mitigate the cross-node memory access wall. Experimental results show that ArcLight significantly surpasses the performance ceiling of mainstream frameworks, achieving up to 46% higher inference throughput. Moreover, ArcLight maintains compatibility with arbitrary CPU devices. ArcLight is publicly available at https://github.com/OpenBMB/ArcLight.