Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU and SmartNIC

TL;DR

Blink introduces a novel LLM inference architecture that eliminates CPU involvement by leveraging SmartNIC and GPU, significantly enhancing performance and stability under interference.

Contribution

It proposes a CPU-free inference stack for LLMs by offloading request handling to SmartNIC and managing batching and scheduling on GPU, improving efficiency and robustness.

Findings

Outperforms baselines in latency and throughput metrics.

Maintains stable performance under CPU interference.

Reduces energy consumption per token.

Abstract

Large Language Model (LLM) inference is rapidly becoming a core datacenter service, yet current serving stacks keep the host CPU on the critical path for orchestration and token-level control. This makes LLM performance sensitive to CPU interference, undermining application colocation and forcing operators to reserve CPU headroom, leaving substantial capacity unutilized. We introduce Blink, an end-to-end serving architecture that removes the host CPU from the steady-state inference path by redistributing responsibilities across a SmartNIC and a GPU. Blink offloads request handling to the SmartNIC, which delivers inputs directly into GPU memory via RDMA, and replaces host-driven scheduling with a persistent GPU kernel that performs batching, scheduling, and KV-cache management without CPU involvement. Evaluated against TensorRT-LLM, vLLM, and SGLang, Blink outperforms all baselines even in isolation, reducing pre-saturation P99 TTFT by up to 8.47$\times$ and P99 TPOT by up to 3.40$\times$, improving decode throughput by up to 2.1$\times$, and reducing energy per token by up to 48.6$\%$. Under CPU interference, Blink maintains stable performance, while existing systems degrade by up to two orders of magnitude.