The Blind Spot of Adaptation: Quantifying and Mitigating Forgetting in Fine-tuned Driving Models

TL;DR

This paper investigates catastrophic forgetting in vision-language models for autonomous driving, introduces a new dataset and benchmark, and proposes the Drive Expert Adapter to mitigate knowledge loss while maintaining performance.

Contribution

It is the first systematic study of forgetting in driving models, introducing a novel prompt-based adaptation framework that preserves pre-trained knowledge.

Findings

Existing methods cause significant knowledge degradation.

The Drive Expert Adapter improves driving task performance.

DEA effectively mitigates catastrophic forgetting.

Abstract

The integration of Vision-Language Models (VLMs) into autonomous driving promises to solve long-tail scenarios, but this paradigm faces the critical and unaddressed challenge of catastrophic forgetting. The very fine-tuning process used to adapt these models to driving-specific data simultaneously erodes their invaluable pre-trained world knowledge, creating a self-defeating paradox that undermines the core reason for their use. This paper provides the first systematic investigation into this phenomenon. We introduce a new large-scale dataset of 180K scenes, which enables the first-ever benchmark specifically designed to quantify catastrophic forgetting in autonomous driving. Our analysis reveals that existing methods suffer from significant knowledge degradation. To address this, we propose the Drive Expert Adapter (DEA), a novel framework that circumvents this trade-off by shifting adaptation from the weight space to the prompt space. DEA dynamically routes inference through different knowledge experts based on scene-specific cues, enhancing driving-task performance without corrupting the model's foundational parameters. Extensive experiments demonstrate that our approach not only achieves state-of-the-art results on driving tasks but also effectively mitigates catastrophic forgetting, preserving the essential generalization capabilities that make VLMs a transformative force for autonomous systems. Data and model are released at FidelityDrivingBench.