Bridging the Cold-Start Gap: LLM-Powered Synthetic Data Generation for Natural Language Search at Airbnb

TL;DR

This paper introduces a framework using large language models to generate synthetic queries and labels, addressing the cold-start problem in Airbnb's natural language search systems, and demonstrating improved realism and diversity over baselines.

Contribution

The authors propose a novel seed-guided synthetic data generation method leveraging LLMs, enhancing cold-start performance and evaluation for NLP search systems.

Findings

Seed-guided approach achieves 0.66 KL divergence in query length, outperforming InPars baseline.

Approach achieves the lowest attribute type KL divergence of 0.04, better than seed queries.

Synthetic data produces harder evaluation examples, improving model discriminative ability.

Abstract

Deploying natural language search systems presents a critical cold-start challenge: no real user queries to learn linguistic patterns, and no relevance labels to train ranking models. We present a framework for generating synthetic queries and labels using large language models (LLMs), powering model training and evaluation for Airbnb's natural language search. For query generation, we combine contrastive listing pairs from booking sessions with seed queries from user research to balance realism and diversity, enabling a cold-to-warm start transition as real user data becomes available. For label generation, we introduce contrastive generation that produces topicality labels by construction, and Virtual Judge (VJ) labeling for broader coverage. We compare our approach against a no-seed contrastive baseline and an InPars-style baseline. For query length, the InPars baseline produces verbose queries with KL divergence of 12.03 vs. real users; our seed-guided approach achieves 0.66, a 7.5x improvement. For attribute type distributions, our approach achieves the lowest KL divergence (0.04), outperforming even seed queries (0.09). Experiments show our approach produces harder evaluation examples than the no-seed baseline (79% vs. 97% pairwise accuracy), providing discriminative signal for model improvement. We deploy production pipelines generating synthetic examples daily for embedding-based retrieval and ranking evaluation.