Multi-Modal Representation Learning with Self-Adaptive Threshold for Commodity Verification

TL;DR

This paper introduces a multi-modal representation learning approach with a self-adaptive threshold for verifying identical commodities in e-commerce, effectively combining image and text data to improve identification accuracy.

Contribution

It proposes an end-to-end dual-stream network that adaptively adjusts verification thresholds, enhancing multi-modal commodity representation and verification in e-commerce scenarios.

Findings

Achieved third place with an F1 score of 0.8936 in a competitive benchmark.

Demonstrated the effectiveness of self-adaptive thresholds in commodity verification.

Validated the advantages of multi-modal representation fusion.

Abstract

In this paper, we propose a method to identify identical commodities. In e-commerce scenarios, commodities are usually described by both images and text. By definition, identical commodities are those that have identical key attributes and are cognitively identical to consumers. There are two main challenges: 1) The extraction and fusion of multi-modal representation. 2) The ability to verify identical commodities by comparing the similarity between representations and a threshold. To address the above problems, we propose an end-to-end multi-modal representation learning method with self-adaptive threshold. We use a dual-stream network to extract multi-modal commodity embeddings and threshold embeddings separately and then concatenate them to obtain commodity representation. Our method is able to adaptively adjust the threshold according to different commodities while maintaining the indexability of the commodity representation space. We experimentally validate the advantages of self-adaptive threshold and the effectiveness of multimodal representation fusion. Besides, our method achieves third place with an F1 score of 0.8936 on the second task of the CCKS-2022 Knowledge Graph Evaluation for Digital Commerce Competition. Code and pretrained models are available at https://github.com/hanchenchen/CCKS2022-track2-solution.