Hybrid Reasoning Based on Large Language Models for Autonomous Car Driving

TL;DR

This paper explores how large language models can enhance autonomous vehicle decision-making by integrating reasoning over sensor data, images, and driving regulations, especially in challenging conditions like low visibility.

Contribution

It demonstrates the potential of LLMs to perform hybrid reasoning combining arithmetic and common-sense knowledge for autonomous driving tasks.

Findings

LLMs can accurately analyze sensor and image data for vehicle control.

Hybrid reasoning improves decision-making in adverse weather conditions.

LLMs outperform traditional methods in complex driving scenarios.

Abstract

Large Language Models (LLMs) have garnered significant attention for their ability to understand text and images, generate human-like text, and perform complex reasoning tasks. However, their ability to generalize this advanced reasoning with a combination of natural language text for decision-making in dynamic situations requires further exploration. In this study, we investigate how well LLMs can adapt and apply a combination of arithmetic and common-sense reasoning, particularly in autonomous driving scenarios. We hypothesize that LLMs hybrid reasoning abilities can improve autonomous driving by enabling them to analyze detected object and sensor data, understand driving regulations and physical laws, and offer additional context. This addresses complex scenarios, like decisions in low visibility (due to weather conditions), where traditional methods might fall short. We evaluated Large Language Models (LLMs) based on accuracy by comparing their answers with human-generated ground truth inside CARLA. The results showed that when a combination of images (detected objects) and sensor data is fed into the LLM, it can offer precise information for brake and throttle control in autonomous vehicles across various weather conditions. This formulation and answers can assist in decision-making for auto-pilot systems.