KLDrive: Fine-Grained 3D Scene Reasoning for Autonomous Driving based on Knowledge Graph

TL;DR

KLDrive introduces a novel framework combining knowledge graphs and large language models to improve fine-grained reasoning and question answering in autonomous driving, reducing hallucinations and enhancing accuracy.

Contribution

It is the first to integrate a scene fact construction module with an LLM for reliable reasoning in autonomous driving scenarios.

Findings

Achieves 65.04% accuracy on NuScenes-QA

Attains 42.45 SPICE score on GVQA

Reduces hallucinations in factual reasoning tasks

Abstract

Autonomous driving requires reliable reasoning over fine-grained 3D scene facts. Fine-grained question answering over multi-modal driving observations provides a natural way to evaluate this capability, yet existing perception pipelines and driving-oriented large language model (LLM) methods still suffer from unreliable scene facts, hallucinations, opaque reasoning, and heavy reliance on task-specific training. We present KLDrive, the first knowledge-graph-augmented LLM reasoning framework for fine-grained question answering in autonomous driving. KLDrive addresses this problem through designing two tightly coupled components: an energy-based scene fact construction module that consolidates multi-source evidence into a reliable scene knowledge graph, and an LLM agent that performs fact-grounded reasoning over a constrained action space under explicit structural constraints. By combining structured prompting with few-shot in-context exemplars, the framework adapts to diverse reasoning tasks without heavy task-specific fine-tuning. Experiments on two large-scale autonomous-driving QA benchmarks show that KLDrive outperforms prior state-of-the-art methods, achieving the best overall accuracy of 65.04% on NuScenes-QA and the best SPICE score of 42.45 on GVQA. On counting, the most challenging factual reasoning task, it improves over the strongest baseline by 46.01 percentage points, demonstrating substantially reduced hallucinations and the benefit of coupling reliable scene fact construction with explicit reasoning.