Enhancing and Scaling Search Query Datasets for Recommendation Systems

TL;DR

This paper introduces a scalable, data-centric system for improving search query datasets in banking recommendation systems, enhancing intent clarity and addressing cold start issues to boost recommendation accuracy.

Contribution

It presents an integrated system with synthetic query generation, intent disambiguation, and gap analysis, demonstrating significant improvements in dataset quality and recommendation precision.

Findings

Synthetic data shows comparable performance on Clinc150 and significant benefits on Banking77.

Intent disambiguation achieves an F1 score of 0.863, improving intent clarity.

Intent gap analysis recovers up to 71% of latent customer needs.

Abstract

This paper presents a deployed, production-grade system designed to enhance and scale search query datasets for intent-based recommendation systems in digital banking. In real-world environments, the growing volume and complexity of user intents create substantial challenges for data management, resulting in suboptimal recommendations and delayed product onboarding. To overcome these challenges, our approach shifts the focus from model-centric enhancements to automated, data-centric strategies. The proposed system integrates three core modules: Synthetic Query Generation, Intent Disambiguation, and Intent Gap Analysis. Synthetic Query Generation produces diverse and realistic user queries. Our experiments reveal no statistically significant difference when using synthetic data for Clinc150, while Banking77 and a proprietary dataset show significant differences. We dig into the underlying factors driving these variations, demonstrating that our approach effectively alleviates the cold start problem (i.e. the challenge of recommending new products with limited historical data). Intent Disambiguation refines broad and overlapping intent categories into precise subintents, achieving an F1 score of 0.863 $\pm$ 0.127 against expert reannotations and leading to clearer differentiation and more precise recommendation mapping. Meanwhile, Intent Gap Analysis identifies latent customer needs by extracting novel intents from unlabeled queries; recovery rates reach up to 71\% in controlled evaluations. Deployed in a live banking environment, our system demonstrates significant improvements in recommendation precision and operation agility, ultimately delivering enhanced user experiences and strategic business benefits. This work underscores the role of high-quality, scalable data in modern AI-driven applications and advocates a proactive approach to data enhancement as a key driver of value.