A Practical Tensor-Network Compression Pipeline for Production-Scale Large Language Models

TL;DR

This paper introduces Minima, a practical pipeline for compressing large language models using tensor decompositions, significantly reducing memory and increasing inference speed for production deployment.

Contribution

Minima presents a novel, production-ready compression pipeline combining sensitivity estimation, tensor decompositions, and custom kernels to optimize large language models for deployment.

Findings

Reduces peak VRAM from 64 GiB to 40 GiB on Qwen3-32B.

Increases throughput from 40 to 75 tokens/sec with Minima and speculative decoding.

Maintains effectiveness under high concurrency with multiple requests.

Abstract

Large language models are limited in deployment by GPU memory and inference latency. We present Minima, a production compression pipeline that learns where and how to structurally compress a Transformer and turns that compression into real serving gains. Minima trains a lightweight convolutional predictor to estimate layer- and patch-level sensitivity, applies a mixture of Tucker, tensor-train, and tensor-ring decompositions to low-sensitivity regions, performs a short healing fine-tune, and executes the resulting operators with custom Triton and CUDA kernels. The reduced memory footprint enables speculative decoding with a small draft model and a larger verifier. On Qwen3-32B at an 8k-token context window, Minima reduces peak VRAM from 64 GiB to 40 GiB. For a single active request, throughput increases from 40 tokens per second (baseline) to 50 tokens per second (Minima) and 75 tokens per second (Minima with speculative decoding). Under 50 parallel requests, throughput is 34, 44, and 53 tokens per second respectively, showing that Minima remains effective under high concurrency even when speculative decoding gains compress. We position Minima relative to recent tensor-network, low-rank plus quantization, and cross-layer sharing methods, and argue that it is a practical step toward more aggressive structural compression via shared tensor backbones with tiny per-layer adapters.