NoteLLM-2: Multimodal Large Representation Models for Recommendation

TL;DR

NoteLLM-2 introduces a novel multimodal representation framework that enhances visual information integration into large language models for improved recommendation performance, addressing previous limitations in multimodal tasks.

Contribution

The paper presents a new end-to-end fine-tuning approach and two innovative methods for integrating visual data into LLMs, improving multimodal recommendation capabilities.

Findings

Effective multimodal representations achieved through prompt-based and late fusion techniques.

Enhanced recommendation performance demonstrated in extensive online and offline experiments.

Framework successfully balances visual and textual information for better item-to-item recommendations.

Abstract

Large Language Models (LLMs) have demonstrated exceptional proficiency in text understanding and embedding tasks. However, their potential in multimodal representation, particularly for item-to-item (I2I) recommendations, remains underexplored. While leveraging existing Multimodal Large Language Models (MLLMs) for such tasks is promising, challenges arise due to their delayed release compared to corresponding LLMs and the inefficiency in representation tasks. To address these issues, we propose an end-to-end fine-tuning method that customizes the integration of any existing LLMs and vision encoders for efficient multimodal representation. Preliminary experiments revealed that fine-tuned LLMs often neglect image content. To counteract this, we propose NoteLLM-2, a novel framework that enhances visual information. Specifically, we propose two approaches: first, a prompt-based method that segregates visual and textual content, employing a multimodal In-Context Learning strategy to balance focus across modalities; second, a late fusion technique that directly integrates visual information into the final representations. Extensive experiments, both online and offline, demonstrate the effectiveness of our approach. Code is available at https://github.com/Applied-Machine-Learning-Lab/NoteLLM.