Why Smaller Is Slower? Dimensional Misalignment in Compressed LLMs

TL;DR

This paper investigates how parameter compression in large language models can cause dimensional misalignment, leading to slower GPU inference, and proposes GAC, a method to optimize for hardware-aligned dimensions to improve speed.

Contribution

The paper introduces GAC, a full-stack approach that re-selects hardware-aligned dimensions during compression to enhance GPU inference speed without sacrificing model quality.

Findings

GAC achieves 100% alignment in compressed models.

GAC recovers up to 1.5× speedup on Llama-3-8B.

Compressed models with GAC maintain comparable quality.

Abstract

Post-training compression reduces LLM parameter counts but often produces irregular tensor dimensions that degrade GPU performance -- a phenomenon we call \emph{dimensional misalignment}. We present a full-stack analysis tracing root causes at three levels: framework, library, and hardware. The key insight is that model inference becomes slower because the resulting dimensions are unfriendly with the GPU execution stack. For example, compressing Llama-3-8B with activation-aware singular value decomposition (ASVD) has 15\% fewer parameters yet runs no faster than the uncompressed baseline, because 95\% of its dimensions are misaligned. We propose \textbf{GAC} (GPU-Aligned Compression), a new compression paradigm that wraps any dimension-reducing compressor and re-selects hardware-aligned dimensions via multi-choice knapsack optimization under the same parameter budget. We evaluate GAC on Llama-3-8B with ASVD and LLM-Pruner, achieving 100\% alignment and recovering up to 1.5$\times$ speedup while preserving model quality.