Can LLMs plan paths in the real world?

TL;DR

This paper evaluates the ability of large language models to plan real-world paths, revealing their unreliability and highlighting the need for improvements in transparency and error-checking mechanisms.

Contribution

The study systematically tests multiple LLMs in real-world navigation scenarios, providing empirical evidence of their limitations in path planning tasks.

Findings

All tested LLMs made numerous errors in path planning.

LLMs are currently unreliable for real-world navigation.

Future improvements should focus on reality checks and transparency.

Abstract

As large language models (LLMs) increasingly integrate into vehicle navigation systems, understanding their path-planning capability is crucial. We tested three LLMs through six real-world path-planning scenarios in various settings and with various difficulties. Our experiments showed that all LLMs made numerous errors in all scenarios, revealing that they are unreliable path planners. We suggest that future work focus on implementing mechanisms for reality checks, enhancing model transparency, and developing smaller models.