Large Language Models in the Travel Domain: An Industrial Experience

TL;DR

This study evaluates the use of large language models in improving data consistency and reducing hallucinations in property descriptions on a travel booking platform, highlighting trade-offs between model quality and computational resources.

Contribution

It compares two LLMs in an industrial setting, demonstrating their effectiveness and resource requirements for enhancing accommodation data quality.

Findings

Mixtral 8x7B achieved higher completeness and precision than Mistral 7B.

Mixtral 8x7B had a lower hallucination rate (1.2%) compared to Mistral 7B (4%).

Resource costs for Mixtral 8x7B were significantly higher than for Mistral 7B.

Abstract

Online property booking platforms are widely used and rely heavily on consistent, up-to-date information about accommodation facilities, often sourced from third-party providers. However, these external data sources are frequently affected by incomplete or inconsistent details, which can frustrate users and result in a loss of market. In response to these challenges, we present an industrial case study involving the integration of Large Language Models (LLMs) into CALEIDOHOTELS, a property reservation platform developed by FERVENTO. We evaluate two well-known LLMs in this context: Mistral 7B, fine-tuned with QLoRA, and Mixtral 8x7B, utilized with a refined system prompt. Both models were assessed based on their ability to generate consistent and homogeneous descriptions while minimizing hallucinations. Mixtral 8x7B outperformed Mistral 7B in terms of completeness (99.6% vs. 93%), precision (98.8% vs. 96%), and hallucination rate (1.2% vs. 4%), producing shorter yet more concise content (249 vs. 277 words on average). However, this came at a significantly higher computational cost: 50GB VRAM and $1.61/hour versus 5GB and $0.16/hour for Mistral 7B. Our findings provide practical insights into the trade-offs between model quality and resource efficiency, offering guidance for deploying LLMs in production environments and demonstrating their effectiveness in enhancing the consistency and reliability of accommodation data.