Hybrid JIT-CUDA Graph Optimization for Low-Latency Large Language Model Inference

TL;DR

This paper introduces a hybrid JIT-CUDA Graph framework that significantly reduces inference latency and variance for large language models, especially in short-sequence, interactive settings.

Contribution

It combines static CUDA Graphs with dynamic JIT kernels to optimize transformer inference, balancing flexibility and performance.

Findings

Reduces Time-to-First-Token by up to 66%.

Lower P99 latency compared to TensorRT-LLM.

Effective for short-sequence LLM workloads.

Abstract

Large Language Models (LLMs) have achieved strong performance across natural language and multimodal tasks, yet their practical deployment remains constrained by inference latency and kernel launch overhead, particularly in interactive, short-sequence settings. This paper presents a hybrid runtime framework that combines Just-In-Time (JIT) compilation with CUDA Graph execution to reduce launch overhead while preserving runtime flexibility during autoregressive decoding. The framework partitions transformer inference into static components executed via CUDA Graph replay and dynamic components handled through JIT-compiled kernels, enabling asynchronous graph capture and reuse across decoding steps. We evaluate the proposed approach on LLaMA-2 7B using single-GPU, batch-size-one inference across prompt lengths from 10 to 500 tokens. Experimental results show that the hybrid runtime reduces Time-to-First-Token (TTFT) by up to 66.0% and achieves lower P99 latency compared with TensorRT-LLM in this regime. These results indicate that hybrid JIT-CUDA Graph execution can effectively reduce inference latency and variance for short-sequence LLM workloads, making it a practical optimization strategy for latency-sensitive AI applications.