Overview of the TREC 2023 Product Product Search Track

TL;DR

This paper introduces the TREC 2023 Product Search Track, focusing on creating a reusable dataset and evaluating how metadata and multi-modal data influence retrieval accuracy in product search, revealing traditional systems' effectiveness and unexpected results for dense retrieval methods.

Contribution

It presents the first evaluation of product search with a new corpus, analyzing the impact of metadata and multi-modal data on retrieval performance, and highlighting the limitations of dense retrieval models in this domain.

Findings

Traditional retrieval systems outperform pretrained embedding models.

No clear benefit from expanded metadata-enhanced collections.

Single-stage dense retrieval often underperforms in product search.

Abstract

This is the first year of the TREC Product search track. The focus this year was the creation of a reusable collection and evaluation of the impact of the use of metadata and multi-modal data on retrieval accuracy. This year we leverage the new product search corpus, which includes contextual metadata. Our analysis shows that in the product search domain, traditional retrieval systems are highly effective and commonly outperform general-purpose pretrained embedding models. Our analysis also evaluates the impact of using simplified and metadata-enhanced collections, finding no clear trend in the impact of the expanded collection. We also see some surprising outcomes; despite their widespread adoption and competitive performance on other tasks, we find single-stage dense retrieval runs can commonly be noncompetitive or generate low-quality results both in the zero-shot and fine-tuned domain.