ThinkDrive: Chain-of-Thought Guided Progressive Reinforcement Learning Fine-Tuning for Autonomous Driving

TL;DR

ThinkDrive introduces a novel two-stage training framework combining chain-of-thought reasoning with progressive reinforcement learning, significantly improving autonomous driving decision-making and generalization over existing methods.

Contribution

It proposes a CoT-guided progressive RL fine-tuning approach with difficulty-aware adaptive policy optimization for autonomous driving.

Findings

Outperforms strong RL baselines by 1.45%, 1.95%, and 1.01% on different metrics.

A 2B-parameter model trained with ThinkDrive surpasses GPT-4o by 3.28%.

Demonstrates improved decision transparency and generalization in autonomous driving.

Abstract

With the rapid advancement of large language models (LLMs) technologies, their application in the domain of autonomous driving has become increasingly widespread. However, existing methods suffer from unstructured reasoning, poor generalization, and misalignment with human driving intent. While Chain-of-Thought (CoT) reasoning enhances decision transparency, conventional supervised fine-tuning (SFT) fails to fully exploit its potential, and reinforcement learning (RL) approaches face instability and suboptimal reasoning depth. We propose ThinkDrive, a CoT guided progressive RL fine-tuning framework for autonomous driving that synergizes explicit reasoning with difficulty-aware adaptive policy optimization. Our method employs a two-stage training strategy. First, we perform SFT using CoT explanations. Then, we apply progressive RL with a difficulty-aware adaptive policy optimizer that dynamically adjusts learning intensity based on sample complexity. We evaluate our approach on a public dataset. The results show that ThinkDrive outperforms strong RL baselines by 1.45%, 1.95%, and 1.01% on exam, easy-exam, and accuracy, respectively. Moreover, a 2B-parameter model trained with our method surpasses the much larger GPT-4o by 3.28% on the exam metric.