ODMA: On-Demand Memory Allocation Strategy for LLM Serving on LPDDR-Class Accelerators

TL;DR

ODMA is a dynamic memory allocation strategy designed for LPDDR accelerators that improves memory utilization and throughput in large language model serving by addressing distribution drift and heavy-tailed request patterns.

Contribution

It introduces a predictor-based adaptive bucket partitioning and safety pool mechanism tailored for LPDDR hardware, overcoming static allocation limitations.

Findings

ODMA increases KV-cache utilization by up to 19.25%.

ODMA improves throughput (TPS) by 23-27%.

ODMA enhances prediction accuracy from 98.60% to 99.55%. on Alpaca.

Abstract

Existing memory management techniques severely hinder efficient Large Language Model serving on accelerators constrained by poor random-access bandwidth.While static pre-allocation preserves memory contiguity,it incurs significant overhead due to worst-case provisioning.Conversely,fine-grained paging mitigates this overhead but relies on HBM's high random-access tolerance, making it unsuitable for LPDDR systems where non-sequential access rapidly degrades bandwidth. Furthermore, prior works typically assume static distributions and HBM characteristics, thereby failing to resolve the critical fragmentation and bandwidth constraints inherent to LPDDR hardware. We present ODMA, an on-demand memory allocation strategy tailored for random-access-constrained accelerators, such as the Cambricon MLU series.ODMA advances generation-length prediction by addressing two critical limitations in production workloads: (i) distribution drift that invalidates static bucket boundaries, and (ii) performance fragility under heavy-tailed request patterns. ODMA integrates a lightweight length predictor with adaptive bucket partitioning and a fallback safety pool. Bucket boundaries are dynamically recalibrated via online histograms to maximize utilization, while the safety pool ensures robustness against prediction errors. On Alpaca and Google-NQ benchmarks, ODMA improves S3's prediction accuracy from 98.60% to 99.55% and 82.68% to 93.36%, respectively. Deployment with DeepSeek-R1-Distill-Qwen-7B on Cambricon MLU370-X4 accelerators demonstrates that ODMA increases KV-cache utilization by up to 19.25% (absolute) and throughput (TPS) by 23-27% over static baselines, validating the efficacy of predictor-driven contiguous allocation for LPDDR-class devices.