Shop The Look: Building a Large Scale Visual Shopping System at Pinterest

TL;DR

This paper describes the development and deployment of 'Shop The Look,' a large-scale visual shopping system at Pinterest that enhances product discovery through visual search, achieving significant improvements in relevance and engagement.

Contribution

The paper presents a comprehensive system for visual shopping at scale, including technology, infrastructure, and data labeling strategies, with real-world deployment and measurable impact.

Findings

Over 160% improvement in human relevance judgments

Over 80% increase in user engagement

Successful deployment in Pinterest production environment

Abstract

As online content becomes ever more visual, the demand for searching by visual queries grows correspondingly stronger. Shop The Look is an online shopping discovery service at Pinterest, leveraging visual search to enable users to find and buy products within an image. In this work, we provide a holistic view of how we built Shop The Look, a shopping oriented visual search system, along with lessons learned from addressing shopping needs. We discuss topics including core technology across object detection and visual embeddings, serving infrastructure for realtime inference, and data labeling methodology for training/evaluation data collection and human evaluation. The user-facing impacts of our system design choices are measured through offline evaluations, human relevance judgements, and online A/B experiments. The collective improvements amount to cumulative relative gains of over 160% in end-to-end human relevance judgements and over 80% in engagement. Shop The Look is deployed in production at Pinterest.