From Pixels to Purchase: Building and Evaluating a Taxonomy-Decoupled Visual Search Engine for Home Goods E-commerce
Cheng Lyu, Jingyue Zhang, Ryan Maunu, Mengwei Li, Vinny DeGenova, Yuanli Pei

TL;DR
This paper introduces a flexible, taxonomy-decoupled visual search engine for home goods e-commerce that improves retrieval quality and customer engagement by using classification-free proposals and an LLM-based evaluation framework.
Contribution
It presents a novel taxonomy-decoupled architecture and an LLM-as-a-Judge evaluation method, enhancing robustness and scalability in visual search systems.
Findings
Improved retrieval quality and customer engagement in a real-world deployment.
Strong correlation between offline metrics and actual business outcomes.
Effective zero-shot evaluation of visual similarity and category relevance.
Abstract
Visual search is critical for e-commerce, especially in style-driven domains where user intent is subjective and open-ended. Existing industrial systems typically couple object detection with taxonomy-based classification and rely on catalog data for evaluation, which is prone to noise that limits robustness and scalability. We propose a taxonomy-decoupled architecture that uses classification-free region proposals and unified embeddings for similarity retrieval, enabling a more flexible and generalizable visual search. To overcome the evaluation bottleneck, we propose an LLM-as-a-Judge framework that assesses nuanced visual similarity and category relevance for query-result pairs in a zero-shot manner, removing dependence on human annotations or noise-prone catalog data. Deployed at scale on a global home goods platform, our system improves retrieval quality and yields a measurable…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Image and Video Retrieval Techniques · Image Retrieval and Classification Techniques · Multimodal Machine Learning Applications
