ChatGPT for GTFS: Benchmarking LLMs on GTFS Understanding and Retrieval

TL;DR

This paper benchmarks ChatGPT's ability to understand and retrieve information from GTFS transit data, revealing moderate comprehension and high accuracy in information extraction with advanced techniques.

Contribution

It evaluates the performance of GPT-3.5-Turbo and GPT-4 on GTFS understanding and retrieval tasks, highlighting the effectiveness of program synthesis over zero-shot methods.

Findings

GPT-4 achieves 73.3% accuracy on MCQs

Program synthesis improves accuracy up to 93% for simple queries

GPT-4 outperforms GPT-3.5-Turbo in information extraction

Abstract

The General Transit Feed Specification (GTFS) standard for publishing transit data is ubiquitous. GTFS being tabular data, with information spread across different files, necessitates specialized tools or packages to retrieve information. Concurrently, the use of Large Language Models(LLMs) for text and information retrieval is growing. The idea of this research is to see if the current widely adopted LLMs (ChatGPT) are able to understand GTFS and retrieve information from GTFS using natural language instructions without explicitly providing information. In this research, we benchmark OpenAI's GPT-3.5-Turbo and GPT-4 LLMs which are the backbone of ChatGPT. ChatGPT demonstrates a reasonable understanding of GTFS by answering 59.7% (GPT-3.5-Turbo) and 73.3% (GPT-4) of our multiple-choice questions (MCQ) correctly. Furthermore, we evaluated the LLMs on information extraction tasks using a filtered GTFS feed containing four routes. We found that program synthesis techniques outperformed zero-shot approaches, achieving up to 93% (90%) accuracy for simple queries and 61% (41%) for complex ones using GPT-4 (GPT-3.5-Turbo).