ProfInfer: An eBPF-based Fine-Grained LLM Inference Profiler

TL;DR

ProfInfer is an eBPF-based, non-intrusive profiling framework that provides detailed, real-time insights into LLM inference engines' performance, aiding optimization and deployment decisions.

Contribution

It introduces a novel, fine-grained profiling system for LLM inference that operates without source modification, offering high fidelity and low overhead.

Findings

Enables operator-level visibility in LLM inference systems

Reveals resource bottlenecks and behavior of complex inference features

Achieves less than 4% runtime overhead in profiling

Abstract

As large language models (LLMs) move from research to production, understanding how inference engines behave in real time has become both essential and elusive. Unlike general-purpose engines such as ONNX Runtime, today's LLM inference systems offer little operator-level visibility, leaving developers blind to where time and resources go. Even basic questions -- is this workload memory-bound or compute-bound? -- often remain unanswered. To close this gap, we develop a fine-grained, non-intrusive profiling framework for modern LLM inference engines, exemplified by llama-cpp but applicable to similar runtime architectures. Built on extended Berkeley Packet Filter (eBPF) technology, our system dynamically attaches probes to runtime functions across multiple layers -- without modifying or recompiling the source. It transforms collected traces into rich visualizations of operators, graphs, timelines, and hardware counter trends, exposing how dense inference, Mixture-of-Experts routing, and operator offloading behave in practice. With less than 4% runtime overhead and high profiling fidelity, our framework makes LLM inference both transparent and diagnosable, turning performance profiling into a practical tool for optimization, scheduling, and resource-aware deployment.