Drive-KD: Multi-Teacher Distillation for VLMs in Autonomous Driving

TL;DR

Drive-KD introduces a multi-teacher knowledge distillation framework for autonomous driving VLMs, significantly reducing memory and latency while maintaining or improving performance across perception, reasoning, and planning tasks.

Contribution

The paper proposes a novel multi-teacher distillation approach with layer-specific attention and asymmetric gradient projection for autonomous driving VLMs, enhancing efficiency and performance.

Findings

Distilled InternVL3-1B outperforms larger models in accuracy and efficiency.

Method generalizes across diverse model architectures and scales.

Achieves higher throughput with substantially less GPU memory.

Abstract

Autonomous driving is an important and safety-critical task, and recent advances in LLMs/VLMs have opened new possibilities for reasoning and planning in this domain. However, large models demand substantial GPU memory and exhibit high inference latency, while conventional supervised fine-tuning (SFT) often struggles to bridge the capability gaps of small models. To address these limitations, we propose Drive-KD, a framework that decomposes autonomous driving into a "perception-reasoning-planning" triad and transfers these capabilities via knowledge distillation. We identify layer-specific attention as the distillation signal to construct capability-specific single-teacher models that outperform baselines. Moreover, we unify these single-teacher settings into a multi-teacher distillation framework and introduce asymmetric gradient projection to mitigate cross-capability gradient conflicts. Extensive evaluations validate the generalization of our method across diverse model families and scales. Experiments show that our distilled InternVL3-1B model, with ~42 times less GPU memory and ~11.4 times higher throughput, achieves better overall performance than the pretrained 78B model from the same family on DriveBench, and surpasses GPT-5.1 on the planning dimension, providing insights toward efficient autonomous driving VLMs.