HAMburger: Accelerating LLM Inference via Token Smashing

TL;DR

HAMburger introduces a novel hierarchical approach to LLM inference that reduces resource usage and increases speed by generating multiple tokens simultaneously and trusting self-drafted tokens, significantly improving efficiency.

Contribution

The paper presents HAMburger, a hierarchical model that redefines resource allocation in LLM inference, enabling multi-token generation per step and reducing computation and memory costs.

Findings

KV cache growth is sub-linear with output length.

Up to 2× reduction in KV cache computation.

Up to 2× increase in throughput (TPS).

Abstract

The growing demand for efficient Large Language Model (LLM) inference requires a holistic optimization on algorithms, systems, and hardware. However, very few works have fundamentally changed the generation pattern: each token needs one forward pass and one KV cache. This can be sub-optimal because we found that LLMs are extremely capable of self-identifying the exact dose of information that a single KV cache can store, and many tokens can be generated confidently without global context. Based on this insight, we introduce HAMburger, a Hierarchically Auto-regressive Model that redefines resource allocation in LLMs by moving beyond uniform computation and storage per token during inference. Stacking a compositional embedder and a micro-step decoder in between a base LLM, HAMburger smashes multiple tokens into a single KV and generates several tokens per step. Additionally, HAMburger functions as a speculative decoding framework where it can blindly trust self-drafted tokens. As a result, HAMburger shifts the growth of KV cache and forward FLOPs from linear to sub-linear with respect to output length, and adjusts its inference speed based on query perplexity and output structure. Extensive evaluations show that HAMburger reduces the KV cache computation by up to 2$\times$ and achieves up to 2$\times$ TPS, while maintaining quality in both short- and long-context tasks. Our method explores an extremely challenging inference regime that requires both computation- and memory-efficiency with a hardware-agnostic design.