Beyond Words: Evaluating Large Language Models in Transportation Planning

TL;DR

This paper evaluates GPT-4 and Phi-3-mini LLMs for transportation planning tasks, demonstrating GPT-4's superior accuracy and reliability, and exploring their potential to transform urban transportation management.

Contribution

It introduces a transportation-informed evaluation framework for LLMs and compares GPT-4 and Phi-3-mini in real-world transportation planning scenarios.

Findings

GPT-4 outperforms Phi-3-mini in GIS and transportation tasks

Phi-3-mini shows competence in resource-constrained environments

GenAI has transformative potential in urban transportation planning

Abstract

The resurgence and rapid advancement of Generative Artificial Intelligence (GenAI) in 2023 has catalyzed transformative shifts across numerous industry sectors, including urban transportation and logistics. This study investigates the evaluation of Large Language Models (LLMs), specifically GPT-4 and Phi-3-mini, to enhance transportation planning. The study assesses the performance and spatial comprehension of these models through a transportation-informed evaluation framework that includes general geospatial skills, general transportation domain skills, and real-world transportation problem-solving. Utilizing a mixed-methods approach, the research encompasses an evaluation of the LLMs' general Geographic Information System (GIS) skills, general transportation domain knowledge as well as abilities to support human decision-making in the real-world transportation planning scenarios of congestion pricing. Results indicate that GPT-4 demonstrates superior accuracy and reliability across various GIS and transportation-specific tasks compared to Phi-3-mini, highlighting its potential as a robust tool for transportation planners. Nonetheless, Phi-3-mini exhibits competence in specific analytical scenarios, suggesting its utility in resource-constrained environments. The findings underscore the transformative potential of GenAI technologies in urban transportation planning. Future work could explore the application of newer LLMs and the impact of Retrieval-Augmented Generation (RAG) techniques, on a broader set of real-world transportation planning and operations challenges, to deepen the integration of advanced AI models in transportation management practices.