Aug2Search: Enhancing Facebook Marketplace Search with LLM-Generated Synthetic Data Augmentation

TL;DR

Aug2Search leverages LLM-generated synthetic data to significantly improve Facebook Marketplace's embedding-based retrieval, enhancing search relevance and diversity with up to 4% ROC_AUC gains.

Contribution

This work introduces a novel multimodal, multitask synthetic data augmentation framework using GenAI for EBR models, demonstrating its effectiveness in a large-scale social commerce setting.

Findings

Synthetic data generated by Llama models is highly coherent and diverse.

Training solely on synthetic data can outperform original data in EBR tasks.

Up to 4% ROC_AUC improvement achieved with synthetic data augmentation.

Abstract

Embedding-Based Retrieval (EBR) is an important technique in modern search engines, enabling semantic match between search queries and relevant results. However, search logging data on platforms like Facebook Marketplace lacks the diversity and details needed for effective EBR model training, limiting the models' ability to capture nuanced search patterns. To address this challenge, we propose Aug2Search, an EBR-based framework leveraging synthetic data generated by Generative AI (GenAI) models, in a multimodal and multitask approach to optimize query-product relevance. This paper investigates the capabilities of GenAI, particularly Large Language Models (LLMs), in generating high-quality synthetic data, and analyzing its impact on enhancing EBR models. We conducted experiments using eight Llama models and 100 million data points from Facebook Marketplace logs. Our synthetic data generation follows three strategies: (1) generate queries, (2) enhance product listings, and (3) generate queries from enhanced listings. We train EBR models on three different datasets: sampled engagement data or original data ((e.g., "Click" and "Listing Interactions")), synthetic data, and a mixture of both engagement and synthetic data to assess their performance across various training sets. Our findings underscore the robustness of Llama models in producing synthetic queries and listings with high coherence, relevance, and diversity, while maintaining low levels of hallucination. Aug2Search achieves an improvement of up to 4% in ROC_AUC with 100 million synthetic data samples, demonstrating the effectiveness of our approach. Moreover, our experiments reveal that with the same volume of training data, models trained exclusively on synthetic data often outperform those trained on original data only or a mixture of original and synthetic data.