Retrieve, Annotate, Evaluate, Repeat: Leveraging Multimodal LLMs for Large-Scale Product Retrieval Evaluation

TL;DR

This paper presents a framework using multimodal large language models to automate and scale the evaluation of product retrieval systems in e-commerce, achieving quality comparable to human annotations while reducing costs.

Contribution

It introduces a novel multimodal LLM-based approach for large-scale product retrieval evaluation, including generating annotation guidelines and conducting annotations, validated on a real e-commerce platform.

Findings

Comparable annotation quality to humans

Significant reduction in time and cost

Effective for large-scale quality control

Abstract

Evaluating production-level retrieval systems at scale is a crucial yet challenging task due to the limited availability of a large pool of well-trained human annotators. Large Language Models (LLMs) have the potential to address this scaling issue and offer a viable alternative to humans for the bulk of annotation tasks. In this paper, we propose a framework for assessing the product search engines in a large-scale e-commerce setting, leveraging Multimodal LLMs for (i) generating tailored annotation guidelines for individual queries, and (ii) conducting the subsequent annotation task. Our method, validated through deployment on a large e-commerce platform, demonstrates comparable quality to human annotations, significantly reduces time and cost, facilitates rapid problem discovery, and provides an effective solution for production-level quality control at scale.