Multi-Modal Recommendation System with Auxiliary Information

TL;DR

This paper enhances context-aware recommendation systems by integrating multi-modal auxiliary information, leading to significant improvements in prediction accuracy across different user sequence datasets.

Contribution

It introduces a method to incorporate comprehensive auxiliary knowledge using embeddings and transformer architectures, advancing beyond sequential context modeling.

Findings

4% increase in NDCG for long sequences

11% increase in NDCG for short sequences

Statistically significant accuracy improvements

Abstract

Context-aware recommendation systems improve upon classical recommender systems by including, in the modelling, a user's behaviour. Research into context-aware recommendation systems has previously only considered the sequential ordering of items as contextual information. However, there is a wealth of unexploited additional multi-modal information available in auxiliary knowledge related to items. This study extends the existing research by evaluating a multi-modal recommendation system that exploits the inclusion of comprehensive auxiliary knowledge related to an item. The empirical results explore extracting vector representations (embeddings) from unstructured and structured data using data2vec. The fused embeddings are then used to train several state-of-the-art transformer architectures for sequential user-item representations. The analysis of the experimental results shows a statistically significant improvement in prediction accuracy, which confirms the effectiveness of including auxiliary information in a context-aware recommendation system. We report a 4% and 11% increase in the NDCG score for long and short user sequence datasets, respectively.