CPRM: A LLM-based Continual Pre-training Framework for Relevance Modeling in Commercial Search

TL;DR

This paper introduces CPRM, a continual pre-training framework for large language models to improve relevance modeling in commercial search by leveraging domain-specific knowledge, in-context learning, and structured item text.

Contribution

The paper proposes a novel CPRM framework that enhances LLM relevance modeling through continual pre-training with query-item data, in-context learning, and reading comprehension modules.

Findings

Significant performance improvements in offline experiments.

Effective online A/B test results showing relevance enhancement.

Better utilization of structured item information.

Abstract

Relevance modeling between queries and items stands as a pivotal component in commercial search engines, directly affecting the user experience. Given the remarkable achievements of large language models (LLMs) in various natural language processing (NLP) tasks, LLM-based relevance modeling is gradually being adopted within industrial search systems. Nevertheless, foundational LLMs lack domain-specific knowledge and do not fully exploit the potential of in-context learning. Furthermore, structured item text remains underutilized, and there is a shortage in the supply of corresponding queries and background knowledge. We thereby propose CPRM (Continual Pre-training for Relevance Modeling), a framework designed for the continual pre-training of LLMs to address these issues. Our CPRM framework includes three modules: 1) employing both queries and multi-field item to jointly pre-train for enhancing domain knowledge, 2) applying in-context pre-training, a novel approach where LLMs are pre-trained on a sequence of related queries or items, and 3) conducting reading comprehension on items to produce associated domain knowledge and background information (e.g., generating summaries and corresponding queries) to further strengthen LLMs. Results on offline experiments and online A/B testing demonstrate that our model achieves convincing performance compared to strong baselines.