Serendipitous Recommendation with Multimodal LLM

TL;DR

This paper introduces a hierarchical framework combining multimodal large language models with traditional recommendation systems to enhance serendipity and user satisfaction in large-scale platforms.

Contribution

It presents a novel hierarchical approach where MLLMs guide conventional recommenders towards more surprising content, addressing scalability and integration challenges.

Findings

Significant improvement in recommendation serendipity.

Enhanced user satisfaction demonstrated in live platform experiments.

Effective chain-of-thought strategy for discovering novel user interests.

Abstract

Conventional recommendation systems succeed in identifying relevant content but often fail to provide users with surprising or novel items. Multimodal Large Language Models (MLLMs) possess the world knowledge and multimodal understanding needed for serendipity, but their integration into billion-item-scale platforms presents significant challenges. In this paper, we propose a novel hierarchical framework where fine-tuned MLLMs provide high-level guidance to conventional recommendation models, steering them towards more serendipitous suggestions. This approach leverages MLLM strengths in understanding multimodal content and user interests while retaining the efficiency of traditional models for item-level recommendation. This mitigates the complexity of applying MLLMs directly to vast action spaces. We also demonstrate a chain-of-thought strategy enabling MLLMs to discover novel user interests by first understanding video content and then identifying relevant yet unexplored interest clusters. Through live experiments within a commercial short-form video platform serving billions of users, we show that our MLLM-powered approach significantly improves both recommendation serendipity and user satisfaction.