Pixie: A System for Recommending 3+ Billion Items to 200+ Million Users in Real-Time

TL;DR

Pixie is a scalable, real-time graph-based recommender system deployed at Pinterest, capable of processing billions of items and users to deliver highly relevant recommendations with low latency, significantly improving user engagement.

Contribution

The paper introduces Pixie, a novel real-time recommendation system utilizing a large object graph and a new random walk algorithm, achieving high scalability and improved recommendation quality.

Findings

Up to 50% increase in user engagement compared to previous systems.

58% improvement in recommendation relevance through graph pruning.

System handles 1,200 requests/sec with 60ms latency.

Abstract

User experience in modern content discovery applications critically depends on high-quality personalized recommendations. However, building systems that provide such recommendations presents a major challenge due to a massive pool of items, a large number of users, and requirements for recommendations to be responsive to user actions and generated on demand in real-time. Here we present Pixie, a scalable graph-based real-time recommender system that we developed and deployed at Pinterest. Given a set of user-specific pins as a query, Pixie selects in real-time from billions of possible pins those that are most related to the query. To generate recommendations, we develop Pixie Random Walk algorithm that utilizes the Pinterest object graph of 3 billion nodes and 17 billion edges. Experiments show that recommendations provided by Pixie lead up to 50% higher user engagement when compared to the previous Hadoop-based production system. Furthermore, we develop a graph pruning strategy at that leads to an additional 58% improvement in recommendations. Last, we discuss system aspects of Pixie, where a single server executes 1,200 recommendation requests per second with 60 millisecond latency. Today, systems backed by Pixie contribute to more than 80% of all user engagement on Pinterest.