iTIMO: An LLM-empowered Synthesis Dataset for Travel Itinerary Modification

TL;DR

This paper introduces iTIMO, a new dataset for travel itinerary modification created using LLMs, enabling better evaluation and development of adaptive travel recommender systems.

Contribution

It formally defines the itinerary modification task, proposes a dataset construction pipeline, and benchmarks LLMs' capabilities in this domain.

Findings

LLMs can generate diverse itinerary modifications based on specified intents.

The iTIMO dataset reveals limitations of current LLMs in handling complex modifications.

Hybrid metrics effectively evaluate the quality of itinerary perturbations.

Abstract

Addressing itinerary modification is crucial for enhancing the travel experience as it is a frequent requirement during traveling. However, existing research mainly focuses on fixed itinerary planning, leaving modification underexplored due to the scarcity of need-to-modify itinerary data. To bridge this gap, we formally define the itinerary modification task and propose a general pipeline to construct the corresponding dataset, namely iTIMO. This pipeline frames the generation of need-to-modify itinerary data as an intent-driven perturbation task. It instructs large language models to perturb real-world itineraries using three operations: REPLACE, ADD, and DELETE. Each perturbation is grounded in three intents: disruptions of popularity, spatial distance, and category diversity. Furthermore, hybrid evaluation metrics are introduced to ensure perturbation effectiveness. We conduct comprehensive benchmarking on iTIMO to analyze the capabilities and limitations of state-of-the-art LLMs. Overall, iTIMO provides a comprehensive testbed for the modification task, and empowers the evolution of traditional travel recommender systems into adaptive frameworks capable of handling dynamic travel needs. Dataset, code and supplementary materials are available at https://github.com/zelo2/iTIMO.