Fine-tuning Multimodal Large Language Models for Product Bundling

TL;DR

This paper introduces Bundle-MLLM, a fine-tuning framework for multimodal large language models that improves product bundling by integrating diverse data types and optimizing for semantic understanding and pattern recognition.

Contribution

It presents a novel hybrid tokenization and fusion approach, along with a progressive optimization strategy, to enhance LLMs' performance in multimodal product bundling tasks.

Findings

Outperforms state-of-the-art methods on four datasets

Effectively integrates textual, media, and relational data

Enhances semantic understanding for product bundling

Abstract

Recent advances in product bundling have leveraged multimodal information through sophisticated encoders, but remain constrained by limited semantic understanding and a narrow scope of knowledge. Therefore, some attempts employ In-context Learning (ICL) to explore the potential of large language models (LLMs) for their extensive knowledge and complex reasoning abilities. However, these efforts are inadequate in understanding mulitmodal data and exploiting LLMs' knowledge for product bundling. To bridge the gap, we introduce Bundle-MLLM, a novel framework that fine-tunes LLMs through a hybrid item tokenization approach within a well-designed optimization strategy. Specifically, we integrate textual, media, and relational data into a unified tokenization, introducing a soft separation token to distinguish between textual and non-textual tokens. Additionally, a streamlined yet powerful multimodal fusion module is employed to embed all non-textual features into a single, informative token, significantly boosting efficiency. To tailor product bundling tasks for LLMs, we reformulate the task as a multiple-choice question with candidate items as options. We further propose a progressive optimization strategy that fine-tunes LLMs for disentangled objectives: 1) learning bundle patterns and 2) enhancing multimodal semantic understanding specific to product bundling. Extensive experiments on four datasets across two domains demonstrate that our approach outperforms a range of state-of-the-art (SOTA) methods.