Enhancing Taobao Display Advertising with Multimodal Representations: Challenges, Approaches and Insights

TL;DR

This paper presents a two-phase framework for integrating multimodal data into Taobao's display advertising system, addressing challenges of effectiveness and cost, leading to significant performance improvements.

Contribution

It introduces a novel two-phase approach combining multimodal pre-training and integration with ID-based models for industrial recommendation systems.

Findings

Significant performance improvements in Taobao advertising system

Effective and cost-efficient multimodal data integration method

Insights for practitioners on leveraging multimodal data

Abstract

Despite the recognized potential of multimodal data to improve model accuracy, many large-scale industrial recommendation systems, including Taobao display advertising system, predominantly depend on sparse ID features in their models. In this work, we explore approaches to leverage multimodal data to enhance the recommendation accuracy. We start from identifying the key challenges in adopting multimodal data in a manner that is both effective and cost-efficient for industrial systems. To address these challenges, we introduce a two-phase framework, including: 1) the pre-training of multimodal representations to capture semantic similarity, and 2) the integration of these representations with existing ID-based models. Furthermore, we detail the architecture of our production system, which is designed to facilitate the deployment of multimodal representations. Since the integration of multimodal representations in mid-2023, we have observed significant performance improvements in Taobao display advertising system. We believe that the insights we have gathered will serve as a valuable resource for practitioners seeking to leverage multimodal data in their systems.