Short-Form Video Recommendations with Multimodal Embeddings: Addressing Cold-Start and Bias Challenges

TL;DR

This paper explores the challenges of recommending short-form videos, such as cold-start and bias issues, and proposes a multimodal retrieval approach that outperforms traditional methods in an e-commerce setting.

Contribution

It introduces a multimodal vision-language retrieval system tailored for short-form video recommendations, addressing cold-start and bias challenges effectively.

Findings

Multimodal retrieval outperforms supervised learning in online tests.

Leveraging vision-language models mitigates position and duration biases.

Effective for new video experiences with limited interaction data.

Abstract

In recent years, social media users have spent significant amounts of time on short-form video platforms. As a result, established platforms in other domains, such as e-commerce, have begun introducing short-form video content to engage users and increase their time spent on the platform. The success of these experiences is due not only to the content itself but also to a unique UI innovation: instead of offering users a list of choices to click, platforms actively recommend content for users to watch one at a time. This creates new challenges for recommender systems, especially when launching a new video experience. Beyond the limited interaction data, immersive feed experiences introduce stronger position bias due to the UI and duration bias when optimizing for watch-time, as models tend to favor shorter videos. These issues, together with the feedback loop inherent in recommender systems, make it difficult to build effective solutions. In this paper, we highlight the challenges faced when introducing a new short-form video experience and present our experience showing that, even with sufficient video interaction data, it can be more beneficial to leverage a video retrieval system using a fine-tuned multimodal vision-language model to overcome these challenges. This approach demonstrated greater effectiveness compared to conventional supervised learning methods in online experiments conducted on our e-commerce platform.