PIMphony: Overcoming Bandwidth and Capacity Inefficiency in PIM-based Long-Context LLM Inference System

TL;DR

PIMphony is a novel PIM orchestrator that enhances long-context LLM inference efficiency by addressing channel utilization, I/O bottlenecks, and memory waste through three co-designed techniques, significantly boosting throughput.

Contribution

It introduces three innovative techniques—TCP, DCS, and DPA—for systematic optimization of PIM-based long-context LLM inference systems.

Findings

Up to 11.3x throughput improvement on PIM-only systems.

Significant reduction in memory waste and I/O bottlenecks.

Enables efficient deployment of large long-context LLMs.

Abstract

The expansion of long-context Large Language Models (LLMs) creates significant memory system challenges. While Processing-in-Memory (PIM) is a promising accelerator, we identify that it suffers from critical inefficiencies when scaled to long contexts: severe channel underutilization, performance-limiting I/O bottlenecks, and massive memory waste from static KV cache management. In this work, we propose PIMphony, a PIM orchestrator that systematically resolves these issues with three co-designed techniques. First, Token-Centric PIM Partitioning (TCP) ensures high channel utilization regardless of batch size. Second, Dynamic PIM Command Scheduling (DCS) mitigates the I/O bottleneck by overlapping data movement and computation. Finally, a Dynamic PIM Access (DPA) controller enables dynamic memory management to eliminate static memory waste. Implemented via an MLIR-based compiler and evaluated on a cycle-accurate simulator, PIMphony significantly improves throughput for long-context LLM inference (up to 72B parameters and 1M context length). Our evaluations show performance boosts of up to 11.3x on PIM-only systems and 8.4x on xPU+PIM systems, enabling more efficient deployment of LLMs in real-world long-context applications.