VLM4Rec: Multimodal Semantic Representation for Recommendation with Large Vision-Language Models

TL;DR

VLM4Rec introduces a semantic alignment approach using large vision-language models to improve multimodal recommendation by grounding item content in natural language, outperforming traditional feature fusion methods.

Contribution

The paper proposes a novel semantic representation framework for multimodal recommendation that leverages large vision-language models for better semantic alignment, surpassing feature fusion techniques.

Findings

VLM4Rec outperforms raw visual features in recommendation tasks.

Semantic alignment improves recommendation accuracy over feature fusion.

Representation quality is more crucial than fusion complexity.

Abstract

Multimodal recommendation is commonly framed as a feature fusion problem, where textual and visual signals are combined to better model user preference. However, the effectiveness of multimodal recommendation may depend not only on how modalities are fused, but also on whether item content is represented in a semantic space aligned with preference matching. This issue is particularly important because raw visual features often preserve appearance similarity, while user decisions are typically driven by higher-level semantic factors such as style, material, and usage context. Motivated by this observation, we propose LVLM-grounded Multimodal Semantic Representation for Recommendation (VLM4Rec), a lightweight framework that organizes multimodal item content through semantic alignment rather than direct feature fusion. VLM4Rec first uses a large vision-language model to ground each item image into an explicit natural-language description, and then encodes the grounded semantics into dense item representations for preference-oriented retrieval. Recommendation is subsequently performed through a simple profile-based semantic matching mechanism over historical item embeddings, yielding a practical offline-online decomposition. Extensive experiments on multiple multimodal recommendation datasets show that VLM4Rec consistently improves performance over raw visual features and several fusion-based alternatives, suggesting that representation quality may matter more than fusion complexity in this setting. The code is released at https://github.com/tyvalencia/enhancing-mm-rec-sys.