Multimodal Item Scoring for Natural Language Recommendation via Gaussian Process Regression with LLM Relevance Judgments

TL;DR

This paper introduces GPR-LLM, a Gaussian Process Regression-based method that leverages LLM relevance judgments to model complex, multimodal relevance functions in Natural Language Recommendation, outperforming existing unimodal approaches.

Contribution

The paper proposes GPR-LLM, a novel approach that uses Gaussian Process Regression with LLM relevance judgments to better capture multimodal relevance distributions in NLRec.

Findings

GPR-LLM with RBF kernel outperforms baseline methods by up to 65%.

Modeling multimodal relevance improves recommendation accuracy.

GPR-LLM is efficient with minimal LLM labeling budget.

Abstract

Natural Language Recommendation (NLRec) generates item suggestions based on the relevance between user-issued NL requests and NL item description passages. Existing NLRec approaches often use Dense Retrieval (DR) to compute item relevance scores from aggregation of inner products between user request embeddings and relevant passage embeddings. However, DR views the request as the sole relevance label, thus leading to a unimodal scoring function centered on the query embedding that is often a weak proxy for query relevance. To better capture the potential multimodal distribution of the relevance scoring function that may arise from complex NLRec data, we propose GPR-LLM that uses Gaussian Process Regression (GPR) with LLM relevance judgments for a subset of candidate passages. Experiments on four NLRec datasets and two LLM backbones demonstrate that GPR-LLM with an RBF kernel, capable of modeling multimodal relevance scoring functions, consistently outperforms simpler unimodal kernels (dot product, cosine similarity), as well as baseline methods including DR, cross-encoder, and pointwise LLM-based relevance scoring by up to 65%. Overall, GPR-LLM provides an efficient and effective approach to NLRec within a minimal LLM labeling budget.