ChinaTravel: An Open-Ended Travel Planning Benchmark with Compositional Constraint Validation for Language Agents

TL;DR

ChinaTravel introduces a comprehensive benchmark for evaluating language agents in open-ended travel planning, emphasizing compositional constraints, implicit user requirements, and neuro-symbolic approaches.

Contribution

It provides a realistic sandbox, a generalizable DSL, an open-ended dataset, and analysis demonstrating neuro-symbolic agents' potential and challenges.

Findings

Neuro-symbolic agents achieved 37.0% constraint satisfaction on human queries.

Pure neural models are less effective, with a 10x lower constraint satisfaction rate.

The benchmark highlights challenges in compositional generalization for travel planning.

Abstract

Travel planning stands out among real-world applications of \emph{Language Agents} because it couples significant practical demand with a rigorous constraint-satisfaction challenge. However, existing benchmarks primarily operate on a slot-filling paradigm, restricting agents to synthetic queries with pre-defined constraint menus, which fails to capture the open-ended nature of natural language interaction, where user requirements are compositional, diverse, and often implicitly expressed. To address this gap, we introduce \emph{ChinaTravel}, with four key contributions: 1) a practical sandbox aligned with the multi-day, multi-POI travel planning, 2) a compositionally generalizable domain-specific language (DSL) for scalable evaluation, covering feasibility, constraint satisfaction, and preference comparison 3) an open-ended dataset that integrates diverse travel requirements and implicit intent from 1154 human participants, and 4) fine-grained analysis reveal the potential of neuro-symbolic agents in travel planning, achieving a 37.0% constraint satisfaction rate on human queries, a 10 \times improvement over purely neural models, yet highlighting significant challenges in compositional generalization. Overall, ChinaTravel provides a foundation for advancing language agents through compositional constraint validation in complex, real-world planning scenarios. Project Page: https://www.lamda.nju.edu.cn/shaojj/ChinaTravel/index.html