Cross-view Semantic Alignment for Livestreaming Product Recognition

TL;DR

This paper introduces LPR4M, a large-scale multimodal dataset for livestreaming product recognition, and proposes RICE, a model that uses contrastive learning and patch-level feature propagation to improve cross-view semantic alignment.

Contribution

The paper presents a new multimodal dataset LPR4M and a novel RICE model that enhances cross-view semantic alignment for livestreaming product recognition.

Findings

RICE outperforms existing methods in recognition accuracy.

LPR4M dataset covers diverse categories and modalities, reflecting real-world scenarios.

Patch Feature Reconstruction effectively reduces semantic misalignment.

Abstract

Live commerce is the act of selling products online through live streaming. The customer's diverse demands for online products introduce more challenges to Livestreaming Product Recognition. Previous works have primarily focused on fashion clothing data or utilize single-modal input, which does not reflect the real-world scenario where multimodal data from various categories are present. In this paper, we present LPR4M, a large-scale multimodal dataset that covers 34 categories, comprises 3 modalities (image, video, and text), and is 50x larger than the largest publicly available dataset. LPR4M contains diverse videos and noise modality pairs while exhibiting a long-tailed distribution, resembling real-world problems. Moreover, a cRoss-vIew semantiC alignmEnt (RICE) model is proposed to learn discriminative instance features from the image and video views of the products. This is achieved through instance-level contrastive learning and cross-view patch-level feature propagation. A novel Patch Feature Reconstruction loss is proposed to penalize the semantic misalignment between cross-view patches. Extensive experiments demonstrate the effectiveness of RICE and provide insights into the importance of dataset diversity and expressivity. The dataset and code are available at https://github.com/adxcreative/RICE