Towards More Relevant Product Search Ranking Via Large Language Models: An Empirical Study

TL;DR

This study explores leveraging Large Language Models to improve e-commerce product search ranking by decomposing relevance and enhancing label and feature generation, leading to more balanced and relevant search results.

Contribution

The paper introduces a novel approach using LLMs for label and feature generation, with sigmoid transformations to better balance content and engagement relevance in ranking models.

Findings

Improved ranking relevance through LLM-based label generation

Enhanced model balance between content and engagement relevance

Positive online and offline evaluation results

Abstract

Training Learning-to-Rank models for e-commerce product search ranking can be challenging due to the lack of a gold standard of ranking relevance. In this paper, we decompose ranking relevance into content-based and engagement-based aspects, and we propose to leverage Large Language Models (LLMs) for both label and feature generation in model training, primarily aiming to improve the model's predictive capability for content-based relevance. Additionally, we introduce different sigmoid transformations on the LLM outputs to polarize relevance scores in labeling, enhancing the model's ability to balance content-based and engagement-based relevances and thus prioritize highly relevant items overall. Comprehensive online tests and offline evaluations are also conducted for the proposed design. Our work sheds light on advanced strategies for integrating LLMs into e-commerce product search ranking model training, offering a pathway to more effective and balanced models with improved ranking relevance.