Quantization-Aware Distillation for NVFP4 Inference Accuracy Recovery

TL;DR

This paper introduces quantization-aware distillation (QAD), a method that effectively recovers the accuracy of quantized large language and vision-language models, especially in complex multi-stage training pipelines, without requiring full training data.

Contribution

The paper presents QAD, a novel distillation technique that improves quantized model accuracy and stability, particularly for models trained with multi-stage pipelines, surpassing traditional quantization-aware training methods.

Findings

QAD achieves near-BF16 accuracy across multiple models.

It is robust to data quality and coverage issues.

QAD simplifies accuracy recovery in complex training pipelines.

Abstract

This technical report presents quantization-aware distillation (QAD) and our best practices for recovering accuracy of NVFP4-quantized large language models (LLMs) and vision-language models (VLMs). QAD distills a full-precision teacher model into a quantized student model using a KL divergence loss. While applying distillation to quantized models is not a new idea, we observe key advantages of QAD for today's LLMs: 1. It shows remarkable effectiveness and stability for models trained through multi-stage post-training pipelines, including supervised fine-tuning (SFT), reinforcement learning (RL), and model merging, where traditional quantization-aware training (QAT) suffers from engineering complexity and training instability; 2. It is robust to data quality and coverage, enabling accuracy recovery without full training data. We evaluate QAD across multiple post-trained models including AceReason Nemotron, Nemotron 3 Nano, Nemotron Nano V2, Nemotron Nano V2 VL (VLM), and Llama Nemotron Super v1, showing consistent recovery to near-BF16 accuracy.