TransportationGames: Benchmarking Transportation Knowledge of (Multimodal) Large Language Models

TL;DR

TransportationGames is a comprehensive benchmark designed to evaluate multimodal large language models' knowledge and capabilities in transportation, highlighting current strengths and areas needing improvement for real-world applications.

Contribution

The paper introduces TransportationGames, the first detailed benchmark for assessing (M)LLMs' transportation knowledge across memorization, understanding, and application tasks.

Findings

Models perform variably across tasks, with significant room for improvement.

The benchmark reveals gaps in models' transportation domain understanding.

TransportationGames provides a foundation for future research in this area.

Abstract

Large language models (LLMs) and multimodal large language models (MLLMs) have shown excellent general capabilities, even exhibiting adaptability in many professional domains such as law, economics, transportation, and medicine. Currently, many domain-specific benchmarks have been proposed to verify the performance of (M)LLMs in specific fields. Among various domains, transportation plays a crucial role in modern society as it impacts the economy, the environment, and the quality of life for billions of people. However, it is unclear how much traffic knowledge (M)LLMs possess and whether they can reliably perform transportation-related tasks. To address this gap, we propose TransportationGames, a carefully designed and thorough evaluation benchmark for assessing (M)LLMs in the transportation domain. By comprehensively considering the applications in real-world scenarios and referring to the first three levels in Bloom's Taxonomy, we test the performance of various (M)LLMs in memorizing, understanding, and applying transportation knowledge by the selected tasks. The experimental results show that although some models perform well in some tasks, there is still much room for improvement overall. We hope the release of TransportationGames can serve as a foundation for future research, thereby accelerating the implementation and application of (M)LLMs in the transportation domain.