Quant-Trim in Practice: Improved Cross-Platform Low-Bit Deployment on Edge NPUs

TL;DR

Quant-Trim is a training method that produces robust, hardware-neutral low-bit quantization checkpoints for edge NPUs, reducing accuracy gaps and dependency on vendor-specific adjustments across different backends.

Contribution

It introduces a backend-agnostic training approach combining progressive fake quantization and reverse pruning to improve low-bit deployment consistency.

Findings

Reduces FP-to-low-bit accuracy gap across models and tasks.

Decreases reliance on vendor-specific calibration and heuristics.

Improves latency, throughput, and energy efficiency metrics on edge hardware.

Abstract

Specialized edge accelerators rely on low-bit quantization, but vendor compilers differ in scaling, clipping, and kernel support, often as black boxes. The same floating-point (FP) checkpoint can therefore yield inconsistent accuracy across backends, forcing practitioners to tweak flags or refactor models to vendor-friendly operator subsets. We introduce Quant-Trim, a training-phase method that produces a hardware-neutral checkpoint robust to backend and precision choices. It combines progressive fake quantization to align training with the deployed integer grid and reverse pruning to tame outlier-driven scale inflation while preserving learnability. Quant-Trim is agnostic to quantization schemes (symmetric/asymmetric, per-tensor/per-channel, INT8/INT4) and requires no vendor-specific graph changes. Across models and tasks, it narrows the FP-to-low-bit gap, reduces dependence on compiler heuristics/calibration, and avoids per-backend retraining. We report accuracy and edge metrics latency, throughput, energy per inference, and cost under static/dynamic activation scaling and varying operator coverage.