Attention-based sequential recommendation system using multimodal data

TL;DR

This paper introduces an attention-based sequential recommendation system that effectively integrates multimodal data such as images, texts, and categories, demonstrating improved performance on Amazon datasets.

Contribution

It presents a novel multimodal attention fusion approach for sequential recommendation, utilizing pre-trained models and multitask learning to enhance recommendation accuracy.

Findings

Outperforms conventional systems on Amazon datasets

Effective integration of multimodal data improves recommendations

Multitask learning enhances generalization performance

Abstract

Sequential recommendation systems that model dynamic preferences based on a use's past behavior are crucial to e-commerce. Recent studies on these systems have considered various types of information such as images and texts. However, multimodal data have not yet been utilized directly to recommend products to users. In this study, we propose an attention-based sequential recommendation method that employs multimodal data of items such as images, texts, and categories. First, we extract image and text features from pre-trained VGG and BERT and convert categories into multi-labeled forms. Subsequently, attention operations are performed independent of the item sequence and multimodal representations. Finally, the individual attention information is integrated through an attention fusion function. In addition, we apply multitask learning loss for each modality to improve the generalization performance. The experimental results obtained from the Amazon datasets show that the proposed method outperforms those of conventional sequential recommendation systems.