Evaluating Retrieval-Augmented Generation Strategies for Large Language Models in Travel Mode Choice Prediction

TL;DR

This paper investigates how retrieval-augmented large language models can improve travel mode choice prediction accuracy and generalization, surpassing traditional models by integrating empirical data with advanced retrieval strategies.

Contribution

It introduces a modular RAG framework for LLMs in travel prediction and evaluates multiple retrieval strategies across different LLM architectures.

Findings

RAG significantly improves prediction accuracy.

GPT-4o with balanced retrieval and re-ranking achieves 80.8% accuracy.

LLMs outperform traditional statistical and machine learning models.

Abstract

Accurately predicting travel mode choice is essential for effective transportation planning, yet traditional statistical and machine learning models are constrained by rigid assumptions, limited contextual reasoning, and reduced generalizability. This study explores the potential of Large Language Models (LLMs) as a more flexible and context-aware approach to travel mode choice prediction, enhanced by Retrieval-Augmented Generation (RAG) to ground predictions in empirical data. We develop a modular framework for integrating RAG into LLM-based travel mode choice prediction and evaluate four retrieval strategies: basic RAG, RAG with balanced retrieval, RAG with a cross-encoder for re-ranking, and RAG with balanced retrieval and cross-encoder for re-ranking. These strategies are tested across three LLM architectures (OpenAI GPT-4o, o4-mini, and o3) to examine the interaction between model reasoning capabilities and retrieval methods. Using the 2023 Puget Sound Regional Household Travel Survey data, we conduct a series of experiments to evaluate model performance. The results demonstrate that RAG substantially enhances predictive accuracy across a range of models. Notably, the GPT-4o model combined with balanced retrieval and cross-encoder re-ranking achieves the highest accuracy of 80.8%, exceeding that of conventional statistical and machine learning baselines. Furthermore, LLM-based models exhibit superior generalization abilities relative to these baselines. Findings highlight the critical interplay between LLM reasoning capabilities and retrieval strategies, demonstrating the importance of aligning retrieval strategies with model capabilities to maximize the potential of LLM-based travel behavior modeling.