Instruct Large Language Models to Drive like Humans

TL;DR

This paper introduces InstructDriver, a method that uses large language models with explicit human instruction tuning and reasoning modules to improve real-world autonomous driving motion planning, emphasizing interpretability and scalability.

Contribution

It presents a novel approach to align LLMs with human driving logic through instruction tuning and reasoning modules, enhancing interpretability and real-world applicability.

Findings

Effective in real-world closed-loop driving scenarios

Able to incorporate human rules and driving data

Demonstrates improved planning performance

Abstract

Motion planning in complex scenarios is the core challenge in autonomous driving. Conventional methods apply predefined rules or learn from driving data to plan the future trajectory. Recent methods seek the knowledge preserved in large language models (LLMs) and apply them in the driving scenarios. Despite the promising results, it is still unclear whether the LLM learns the underlying human logic to drive. In this paper, we propose an InstructDriver method to transform LLM into a motion planner with explicit instruction tuning to align its behavior with humans. We derive driving instruction data based on human logic (e.g., do not cause collisions) and traffic rules (e.g., proceed only when green lights). We then employ an interpretable InstructChain module to further reason the final planning reflecting the instructions. Our InstructDriver allows the injection of human rules and learning from driving data, enabling both interpretability and data scalability. Different from existing methods that experimented on closed-loop or simulated settings, we adopt the real-world closed-loop motion planning nuPlan benchmark for better evaluation. InstructDriver demonstrates the effectiveness of the LLM planner in a real-world closed-loop setting. Our code is publicly available at https://github.com/bonbon-rj/InstructDriver.