Meta Learning to Rank for Sparsely Supervised Queries

TL;DR

This paper introduces a meta learning framework for learning to rank in scenarios with sparse supervision, enabling models to adapt quickly to new queries with limited labeled data, especially in diverse and privacy-sensitive domains.

Contribution

The paper proposes a novel meta learning to rank approach that adapts to query-specific optimal parameters, improving performance on sparsely supervised queries compared to traditional models.

Findings

Significant performance improvements on public datasets.

Effective adaptation to new, diverse queries.

Robustness in privacy-constrained and expert-labeled scenarios.

Abstract

Supervisory signals are a critical resource for training learning to rank models. In many real-world search and retrieval scenarios, these signals may not be readily available or could be costly to obtain for some queries. The examples include domains where labeling requires professional expertise, applications with strong privacy constraints, and user engagement information that are too scarce. We refer to these scenarios as sparsely supervised queries which pose significant challenges to traditional learning to rank models. In this work, we address sparsely supervised queries by proposing a novel meta learning to rank framework which leverages fast learning and adaption capability of meta-learning. The proposed approach accounts for the fact that different queries have different optimal parameters for their rankers, in contrast to traditional learning to rank models which only learn a global ranking model applied to all the queries. In consequence, the proposed method would yield significant advantages especially when new queries are of different characteristics with the training queries. Moreover, the proposed meta learning to rank framework is generic and flexible. We conduct a set of comprehensive experiments on both public datasets and a real-world e-commerce dataset. The results demonstrate that the proposed meta-learning approach can significantly enhance the performance of learning to rank models with sparsely labeled queries.