On the Necessity of World Knowledge for Mitigating Missing Labels in Extreme Classification

TL;DR

This paper introduces SKIM, a scalable method combining small language models and unstructured meta-data to address missing labels in extreme classification, significantly improving retrieval performance and online ad click-yield.

Contribution

The paper presents SKIM, a novel scalable algorithm that leverages small language models and meta-data to mitigate missing labels in large-scale extreme classification tasks.

Findings

SKIM outperforms existing methods on Recall@100 by over 10 points.

SKIM scales to datasets with 10 million documents, outperforming others by 12%.

Online A/B tests show a 1.23% increase in ad click-yield.

Abstract

Extreme Classification (XC) aims to map a query to the most relevant documents from a very large document set. XC algorithms used in real-world applications learn this mapping from datasets curated from implicit feedback, such as user clicks. However, these datasets inevitably suffer from missing labels. In this work, we observe that systematic missing labels lead to missing knowledge, which is critical for accurately modelling relevance between queries and documents. We formally show that this absence of knowledge cannot be recovered using existing methods such as propensity weighting and data imputation strategies that solely rely on the training dataset. While LLMs provide an attractive solution to augment the missing knowledge, leveraging them in applications with low latency requirements and large document sets is challenging. To incorporate missing knowledge at scale, we propose SKIM (Scalable Knowledge Infusion for Missing Labels), an algorithm that leverages a combination of small LM and abundant unstructured meta-data to effectively mitigate the missing label problem. We show the efficacy of our method on large-scale public datasets through exhaustive unbiased evaluation ranging from human annotations to simulations inspired from industrial settings. SKIM outperforms existing methods on Recall@100 by more than 10 absolute points. Additionally, SKIM scales to proprietary query-ad retrieval datasets containing 10 million documents, outperforming contemporary methods by 12% in offline evaluation and increased ad click-yield by 1.23% in an online A/B test conducted on a popular search engine. We release our code, prompts, trained XC models and finetuned SLMs at: https://github.com/bicycleman15/skim