Spatiotemporal Graph Guided Multi-modal Network for Livestreaming Product Retrieval

TL;DR

This paper introduces SGMN, a novel spatiotemporal graph-based multi-modal network that improves livestreaming product retrieval by leveraging text-guided attention, long-range graph modeling, and hard example mining to address key challenges.

Contribution

The paper presents a new multi-modal network architecture that effectively handles distractors, video-image heterogeneity, and subtle product differences in livestreaming product retrieval.

Findings

SGMN outperforms state-of-the-art methods significantly.

The text-guided attention improves focus on intended products.

Hard example mining enhances fine-grained product discrimination.

Abstract

With the rapid expansion of e-commerce, more consumers have become accustomed to making purchases via livestreaming. Accurately identifying the products being sold by salespeople, i.e., livestreaming product retrieval (LPR), poses a fundamental and daunting challenge. The LPR task encompasses three primary dilemmas in real-world scenarios: 1) the recognition of intended products from distractor products present in the background; 2) the video-image heterogeneity that the appearance of products showcased in live streams often deviates substantially from standardized product images in stores; 3) there are numerous confusing products with subtle visual nuances in the shop. To tackle these challenges, we propose the Spatiotemporal Graphing Multi-modal Network (SGMN). First, we employ a text-guided attention mechanism that leverages the spoken content of salespeople to guide the model to focus toward intended products, emphasizing their salience over cluttered background products. Second, a long-range spatiotemporal graph network is further designed to achieve both instance-level interaction and frame-level matching, solving the misalignment caused by video-image heterogeneity. Third, we propose a multi-modal hard example mining, assisting the model in distinguishing highly similar products with fine-grained features across the video-image-text domain. Through extensive quantitative and qualitative experiments, we demonstrate the superior performance of our proposed SGMN model, surpassing the state-of-the-art methods by a substantial margin. The code is available at https://github.com/Huxiaowan/SGMN.