ZhichunRoad at Amazon KDD Cup 2022: MultiTask Pre-Training for E-Commerce Product Search

TL;DR

This paper introduces a multilingual multi-task pre-trained model for e-commerce search, combining various training strategies to enhance robustness and generalization, achieving top-8 rankings in KDD Cup 2022 tasks.

Contribution

It presents a novel multi-task pre-training framework with advanced fine-tuning techniques and semantic enhancements for improved e-commerce search performance.

Findings

Achieved top-8 ranking in three KDD Cup 2022 tasks.

Demonstrated improved robustness and generalization through multi-task pre-training.

Utilized multi-granular semantic units for better textual representation.

Abstract

In this paper, we propose a robust multilingual model to improve the quality of search results. Our model not only leverage the processed class-balanced dataset, but also benefit from multitask pre-training that leads to more general representations. In pre-training stage, we adopt mlm task, classification task and contrastive learning task to achieve considerably performance. In fine-tuning stage, we use confident learning, exponential moving average method (EMA), adversarial training (FGM) and regularized dropout strategy (R-Drop) to improve the model's generalization and robustness. Moreover, we use a multi-granular semantic unit to discover the queries and products textual metadata for enhancing the representation of the model. Our approach obtained competitive results and ranked top-8 in three tasks. We release the source code and pre-trained models associated with this work.