Multimodal Generative Recommendation for Fusing Semantic and Collaborative Signals

TL;DR

This paper introduces MSCGRec, a multimodal generative recommender that fuses semantic and collaborative signals, employing self-supervised image quantization and constrained sequence learning to outperform existing methods on large datasets.

Contribution

The paper proposes MSCGRec, a novel multimodal generative recommender integrating semantic modalities and collaborative signals with a new constrained sequence learning approach.

Findings

Outperforms baseline methods on three large datasets

Effective fusion of semantic and collaborative signals

Validated through extensive ablation studies

Abstract

Sequential recommender systems rank relevant items by modeling a user's interaction history and computing the inner product between the resulting user representation and stored item embeddings. To avoid the significant memory overhead of storing large item sets, the generative recommendation paradigm instead models each item as a series of discrete semantic codes. Here, the next item is predicted by an autoregressive model that generates the code sequence corresponding to the predicted item. However, despite promising ranking capabilities on small datasets, these methods have yet to surpass traditional sequential recommenders on large item sets, limiting their adoption in the very scenarios they were designed to address. To resolve this, we propose MSCGRec, a Multimodal Semantic and Collaborative Generative Recommender. MSCGRec incorporates multiple semantic modalities and introduces a novel self-supervised quantization learning approach for images based on the DINO framework. Additionally, MSCGRec fuses collaborative and semantic signals by extracting collaborative features from sequential recommenders and treating them as a separate modality. Finally, we propose constrained sequence learning that restricts the large output space during training to the set of permissible tokens. We empirically demonstrate on three large real-world datasets that MSCGRec outperforms both sequential and generative recommendation baselines and provide an extensive ablation study to validate the impact of each component.