QARM: Quantitative Alignment Multi-Modal Recommendation at Kuaishou

TL;DR

This paper introduces QARM, a multi-modal recommendation framework that aligns and customizes multi-modal representations for improved user interest modeling in industry settings.

Contribution

It proposes a novel quantitative multi-modal framework that addresses representation unmatching and unlearning issues, enabling trainable and task-specific multi-modal representations.

Findings

Improved recommendation accuracy with aligned multi-modal representations.

Effective customization of multi-modal info for downstream tasks.

Addresses key limitations of pre-trained multi-modal models.

Abstract

In recent years, with the significant evolution of multi-modal large models, many recommender researchers realized the potential of multi-modal information for user interest modeling. In industry, a wide-used modeling architecture is a cascading paradigm: (1) first pre-training a multi-modal model to provide omnipotent representations for downstream services; (2) The downstream recommendation model takes the multi-modal representation as additional input to fit real user-item behaviours. Although such paradigm achieves remarkable improvements, however, there still exist two problems that limit model performance: (1) Representation Unmatching: The pre-trained multi-modal model is always supervised by the classic NLP/CV tasks, while the recommendation models are supervised by real user-item interaction. As a result, the two fundamentally different tasks' goals were relatively separate, and there was a lack of consistent objective on their representations; (2) Representation Unlearning: The generated multi-modal representations are always stored in cache store and serve as extra fixed input of recommendation model, thus could not be updated by recommendation model gradient, further unfriendly for downstream training. Inspired by the two difficulties challenges in downstream tasks usage, we introduce a quantitative multi-modal framework to customize the specialized and trainable multi-modal information for different downstream models.