To Judge or not to Judge: Using LLM Judgements for Advertiser Keyphrase Relevance at eBay

TL;DR

This paper explores using large language models as scalable proxies for human seller judgment to improve keyphrase relevance in e-commerce advertising, aligning machine predictions with human preferences and business metrics.

Contribution

It introduces a novel approach of leveraging LLM judgments to better align relevance models with human seller perceptions in eBay's advertising ecosystem.

Findings

LLM judgments improve relevance model alignment with seller preferences.

Using LLMs as judges enhances harmony among seller, advertising, and search systems.

A rigorous evaluation framework grounded in business metrics is essential.

Abstract

E-commerce sellers are recommended keyphrases based on their inventory on which they advertise to increase buyer engagement (clicks/sales). The relevance of advertiser keyphrases plays an important role in preventing the inundation of search systems with numerous irrelevant items that compete for attention in auctions, in addition to maintaining a healthy seller perception. In this work, we describe the shortcomings of training Advertiser keyphrase relevance filter models on click/sales/search relevance signals and the importance of aligning with human judgment, as sellers have the power to adopt or reject said keyphrase recommendations. In this study, we frame Advertiser keyphrase relevance as a complex interaction between 3 dynamical systems -- seller judgment, which influences seller adoption of our product, Advertising, which provides the keyphrases to bid on, and Search, who holds the auctions for the same keyphrases. This study discusses the practicalities of using human judgment via a case study at eBay Advertising and demonstrate that using LLM-as-a-judge en-masse as a scalable proxy for seller judgment to train our relevance models achieves a better harmony across the three systems -- provided that they are bound by a meticulous evaluation framework grounded in business metrics.