MCAP: Deployment-Time Layer Profiling for Memory-Constrained LLM Inference

TL;DR

MCAP is a load-time profiling method that dynamically optimizes memory and precision allocation for large language models on heterogeneous hardware, improving throughput and enabling models to run within constrained memory budgets.

Contribution

Introduces MCAP, a lightweight per-layer importance estimator for dynamic precision and memory placement in LLM deployment, enhancing performance and memory efficiency.

Findings

Achieves 1.5-1.8x higher decode throughput on NVIDIA T4.

Enables models to operate in previously infeasible memory regimes.

Allows a single set of weights to adapt across diverse memory budgets.

Abstract

Deploying large language models to heterogeneous hardware is often constrained by memory, not compute. We introduce MCAP (Monte Carlo Activation Profiling), a load-time per-layer importance estimator that enables dynamic precision and memory placement decisions on the target device. MCAP produces a lightweight per-layer signal that drives both precision dispatch (W4A8 vs. W4A16) and residency tier (GPU, RAM, SSD), allowing a single set of weights to operate across diverse memory budgets. Our system, NVE, achieves 1.5-1.8x higher decode throughput than llama-cpp Q4_0 on NVIDIA T4 and enables models to run in memory regimes previously infeasible without modifying weights.