Visual Product Graph: Bridging Visual Products And Composite Images For End-to-End Style Recommendations

TL;DR

This paper introduces the Visual Product Graph (VPG), a real-time system that connects individual products to composite images for enhanced style recommendations and visual search, improving relevance and engagement.

Contribution

The paper presents the design and implementation of VPG, integrating advanced computer vision models for object detection and visual embeddings to enable end-to-end style recommendations.

Findings

Achieved 78.8% extremely similar@1 in relevance evaluations

Real-time retrieval system with 6% module engagement rate

Deployed in production at Pinterest

Abstract

Retrieving semantically similar but visually distinct contents has been a critical capability in visual search systems. In this work, we aim to tackle this problem with Visual Product Graph (VPG), leveraging high-performance infrastructure for storage and state-of-the-art computer vision models for image understanding. VPG is built to be an online real-time retrieval system that enables navigation from individual products to composite scenes containing those products, along with complementary recommendations. Our system not only offers contextual insights by showcasing how products can be styled in a context, but also provides recommendations for complementary products drawn from these inspirations. We discuss the essential components for building the Visual Product Graph, along with the core computer vision model improvements across object detection, foundational visual embeddings, and other visual signals. Our system achieves a 78.8% extremely similar@1 in end-to-end human relevance evaluations, and a 6% module engagement rate. The "Ways to Style It" module, powered by the Visual Product Graph technology, is deployed in production at Pinterest.