Boosting LLM-based Relevance Modeling with Distribution-Aware Robust Learning

TL;DR

This paper introduces DaRL, a novel framework that enhances LLM-based relevance modeling by improving fine-grained discrimination and robustness to data distribution shifts, specifically applied to Alipay Search.

Contribution

The paper proposes a distribution-aware robust learning framework with a new loss function, sample augmentation, and multi-stage fine-tuning to improve relevance modeling performance and generalization.

Findings

DaRL improves relevance discrimination across fine-grained relevance levels.

DaRL enhances model robustness to distribution shifts in real-world scenarios.

DaRL achieves better online performance in Alipay Search.

Abstract

With the rapid advancement of pre-trained large language models (LLMs), recent endeavors have leveraged the capabilities of LLMs in relevance modeling, resulting in enhanced performance. This is usually done through the process of fine-tuning LLMs on specifically annotated datasets to determine the relevance between queries and items. However, there are two limitations when LLMs are naively employed for relevance modeling through fine-tuning and inference. First, it is not inherently efficient for performing nuanced tasks beyond simple yes or no answers, such as assessing search relevance. It may therefore tend to be overconfident and struggle to distinguish fine-grained degrees of relevance (e.g., strong relevance, weak relevance, irrelevance) used in search engines. Second, it exhibits significant performance degradation when confronted with data distribution shift in real-world scenarios. In this paper, we propose a novel Distribution-Aware Robust Learning framework (DaRL) for relevance modeling in Alipay Search. Specifically, we design an effective loss function to enhance the discriminability of LLM-based relevance modeling across various fine-grained degrees of query-item relevance. To improve the generalizability of LLM-based relevance modeling, we first propose the Distribution-Aware Sample Augmentation (DASA) module. This module utilizes out-of-distribution (OOD) detection techniques to actively select appropriate samples that are not well covered by the original training set for model fine-tuning. Furthermore, we adopt a multi-stage fine-tuning strategy to simultaneously improve in-distribution (ID) and OOD performance, bridging the performance gap between them. DaRL has been deployed online to serve the Alipay's insurance product search...