AdaDrive: Self-Adaptive Slow-Fast System for Language-Grounded Autonomous Driving

TL;DR

AdaDrive is a novel adaptive framework that intelligently balances the use of Large Language Models and traditional planning in autonomous driving, optimizing decision-making accuracy and efficiency in dynamic scenarios.

Contribution

It introduces an adaptive activation and fusion strategy for LLMs in autonomous driving, enabling context-aware collaboration that improves performance and reduces computational overhead.

Findings

Achieves state-of-the-art driving accuracy on benchmarks.

Reduces LLM invocation frequency, saving computational resources.

Maintains real-time performance while leveraging LLMs effectively.

Abstract

Effectively integrating Large Language Models (LLMs) into autonomous driving requires a balance between leveraging high-level reasoning and maintaining real-time efficiency. Existing approaches either activate LLMs too frequently, causing excessive computational overhead, or use fixed schedules, failing to adapt to dynamic driving conditions. To address these challenges, we propose AdaDrive, an adaptively collaborative slow-fast framework that optimally determines when and how LLMs contribute to decision-making. (1) When to activate the LLM: AdaDrive employs a novel adaptive activation loss that dynamically determines LLM invocation based on a comparative learning mechanism, ensuring activation only in complex or critical scenarios. (2) How to integrate LLM assistance: Instead of rigid binary activation, AdaDrive introduces an adaptive fusion strategy that modulates a continuous, scaled LLM influence based on scene complexity and prediction confidence, ensuring seamless collaboration with conventional planners. Through these strategies, AdaDrive provides a flexible, context-aware framework that maximizes decision accuracy without compromising real-time performance. Extensive experiments on language-grounded autonomous driving benchmarks demonstrate that AdaDrive state-of-the-art performance in terms of both driving accuracy and computational efficiency. Code is available at https://github.com/ReaFly/AdaDrive.