Multimodal Generative Retrieval Model with Staged Pretraining for Food Delivery on Meituan

TL;DR

This paper introduces a staged pretraining approach for multimodal retrieval in food delivery, improving feature utilization and training stability, leading to better retrieval accuracy and increased platform revenue.

Contribution

The paper proposes a novel staged pretraining strategy and semantic ID tasks to enhance multimodal feature learning and address training challenges in retrieval models.

Findings

Achieved significant improvements in retrieval metrics (R@5, R@10, R@20, N@5, N@10, N@20).

Demonstrated a 1.12% revenue increase in real-world A/B testing.

Validated the effectiveness of staged pretraining in practical food delivery scenarios.

Abstract

Multimodal retrieval models are becoming increasingly important in scenarios such as food delivery, where rich multimodal features can meet diverse user needs and enable precise retrieval. Mainstream approaches typically employ a dual-tower architecture between queries and items, and perform joint optimization of intra-tower and inter-tower tasks. However, we observe that joint optimization often leads to certain modalities dominating the training process, while other modalities are neglected. In addition, inconsistent training speeds across modalities can easily result in the one-epoch problem. To address these challenges, we propose a staged pretraining strategy, which guides the model to focus on specialized tasks at each stage, enabling it to effectively attend to and utilize multimodal features, and allowing flexible control over the training process at each stage to avoid the one-epoch problem. Furthermore, to better utilize the semantic IDs that compress high-dimensional multimodal embeddings, we design both generative and discriminative tasks to help the model understand the associations between SIDs, queries, and item features, thereby improving overall performance. Extensive experiments on large-scale real-world Meituan data demonstrate that our method achieves improvements of 3.80%, 2.64%, and 2.17% on R@5, R@10, and R@20, and 5.10%, 4.22%, and 2.09% on N@5, N@10, and N@20 compared to mainstream baselines. Online A/B testing on the Meituan platform shows that our approach achieves a 1.12% increase in revenue and a 1.02% increase in click-through rate, validating the effectiveness and superiority of our method in practical applications.