LETToT: Label-Free Evaluation of Large Language Models On Tourism Using Expert Tree-of-Thought

TL;DR

This paper introduces LETToT, a label-free, expert tree-of-thought framework for evaluating large language models in tourism, demonstrating improved accuracy and scalability without relying on annotated data.

Contribution

We propose a novel, scalable, label-free evaluation method using expert reasoning structures, outperforming traditional benchmark approaches in domain-specific LLM assessment.

Findings

Expert ToT achieves 4.99-14.15% quality gains over baselines.

Scaling laws hold in tourism domain, with reasoning models closing the gap.

Explicit reasoning architectures outperform in accuracy and conciseness for smaller models.

Abstract

Evaluating large language models (LLMs) in specific domain like tourism remains challenging due to the prohibitive cost of annotated benchmarks and persistent issues like hallucinations. We propose $\textbf{L}$able-Free $\textbf{E}$valuation of LLM on $\textbf{T}$ourism using Expert $\textbf{T}$ree-$\textbf{o}$f-$\textbf{T}$hought (LETToT), a framework that leverages expert-derived reasoning structures-instead of labeled data-to access LLMs in tourism. First, we iteratively refine and validate hierarchical ToT components through alignment with generic quality dimensions and expert feedback. Results demonstrate the effectiveness of our systematically optimized expert ToT with 4.99-14.15\% relative quality gains over baselines. Second, we apply LETToT's optimized expert ToT to evaluate models of varying scales (32B-671B parameters), revealing: (1) Scaling laws persist in specialized domains (DeepSeek-V3 leads), yet reasoning-enhanced smaller models (e.g., DeepSeek-R1-Distill-Llama-70B) close this gap; (2) For sub-72B models, explicit reasoning architectures outperform counterparts in accuracy and conciseness ($p<0.05$). Our work established a scalable, label-free paradigm for domain-specific LLM evaluation, offering a robust alternative to conventional annotated benchmarks.