LLM-Emu: Native Runtime Emulation of LLM Inference via Profile-Driven Sampling

TL;DR

LLM-Emu is a lightweight, serving-native emulator for LLM inference that accurately mimics real system behavior using profile-driven sampling, enabling cost-effective online evaluation.

Contribution

It introduces LLM-Emu, a novel emulator that preserves production paths while replacing GPU execution with sampled latency, facilitating practical online experimentation.

Findings

LLM-Emu closely tracks real vLLM serving behavior with errors within 5.3%.

It maintains throughput within 1.9% of real systems across diverse workloads.

The emulator is open sourced at https://github.com/AKafakA/llm-emu.

Abstract

Realistic evaluation of LLM serving systems requires online workloads, dynamic arrivals, queueing, and the serving engine's local scheduling for execution batching, but running such experiments on GPUs is expensive. Existing simulators reduce this cost, but often operate offline or in time-warped mode, re-implement serving-engine schedulers, or require accurate operator/kernel-level latency models. We present LLM-Emu, a serving-native emulator for vLLM that preserves the production HTTP, scheduling, KV-cache, and output-processing paths while replacing only GPU forward execution with profile-sampled latency and synthetic output tokens. Tested on two different GPUs, four model variants, two model families, two attention backends, and both Poisson and bursty ShareGPT workloads, LLM-Emu closely tracks real vLLM serving behavior: TPOT and ITL stay within $4.8\%$ absolute error, E2E latency within $5.3\%$, and output throughput within $1.9\%$; TTFT is less stable, with maximum error $10.4\%$, reflecting its sensitivity to admission and queue state. These results suggest that lightweight, serving-native emulation can support practical online experimentation for LLM-serving systems. LLM-Emu is open sourced at https://github.com/AKafakA/llm-emu.