Fine-grained large-scale content recommendations for MSX sellers

TL;DR

This paper introduces a large-scale, fine-grained content recommendation system for Microsoft sellers, focusing on semantic matching at the opportunity level to efficiently surface relevant content from a vast repository.

Contribution

It presents a novel semantic matching model optimized for large-scale, opportunity-specific content recommendations, with extensive evaluation of model architectures and quality assurance methods.

Findings

Achieved accurate top-5 content recommendations out of 40,000 items.

Demonstrated efficient semantic matching over large datasets.

Validated recommendation quality using human experts and LLM-based evaluation.

Abstract

One of the most critical tasks of Microsoft sellers is to meticulously track and nurture potential business opportunities through proactive engagement and tailored solutions. Recommender systems play a central role to help sellers achieve their goals. In this paper, we present a content recommendation model which surfaces various types of content (technical documentation, comparison with competitor products, customer success stories etc.) that sellers can share with their customers or use for their own self-learning. The model operates at the opportunity level which is the lowest possible granularity and the most relevant one for sellers. It is based on semantic matching between metadata from the contents and carefully selected attributes of the opportunities. Considering the volume of seller-managed opportunities in organizations such as Microsoft, we show how to perform efficient semantic matching over a very large number of opportunity-content combinations. The main challenge is to ensure that the top-5 relevant contents for each opportunity are recommended out of a total of $\approx 40,000$ published contents. We achieve this target through an extensive comparison of different model architectures and feature selection. Finally, we further examine the quality of the recommendations in a quantitative manner using a combination of human domain experts as well as by using the recently proposed "LLM as a judge" framework.