CodeMerge: Codebook-Guided Model Merging for Robust Test-Time Adaptation in Autonomous Driving

TL;DR

CodeMerge introduces a lightweight, scalable model merging method operating in a compact latent space, significantly enhancing test-time adaptation robustness for 3D perception in autonomous driving without high computational costs.

Contribution

It proposes a novel latent space-based model merging framework that improves efficiency and stability in test-time adaptation for autonomous driving perception tasks.

Findings

Achieves 14.9% NDS improvement on nuScenes-C

Over 7.6% mAP increase on nuScenes-to-KITTI

Enhances downstream tasks like mapping and motion prediction

Abstract

Maintaining robust 3D perception under dynamic and unpredictable test-time conditions remains a critical challenge for autonomous driving systems. Existing test-time adaptation (TTA) methods often fail in high-variance tasks like 3D object detection due to unstable optimization and sharp minima. While recent model merging strategies based on linear mode connectivity (LMC) offer improved stability by interpolating between fine-tuned checkpoints, they are computationally expensive, requiring repeated checkpoint access and multiple forward passes. In this paper, we introduce CodeMerge, a lightweight and scalable model merging framework that bypasses these limitations by operating in a compact latent space. Instead of loading full models, CodeMerge represents each checkpoint with a low-dimensional fingerprint derived from the source model's penultimate features and constructs a key-value codebook. We compute merging coefficients using ridge leverage scores on these fingerprints, enabling efficient model composition without compromising adaptation quality. Our method achieves strong performance across challenging benchmarks, improving end-to-end 3D detection 14.9% NDS on nuScenes-C and LiDAR-based detection by over 7.6% mAP on nuScenes-to-KITTI, while benefiting downstream tasks such as online mapping, motion prediction and planning even without training. Code and pretrained models are released in the supplementary material.