Dataset and Models for Item Recommendation Using Multi-Modal User Interactions

TL;DR

This paper introduces a new dataset and models for item recommendation using multi-modal user interactions across different channels, addressing the challenge of incomplete modalities and demonstrating the effectiveness of a novel approach that maps interactions to a common feature space.

Contribution

The paper provides a real-world dataset for multi-modal user interactions and proposes a new method to handle missing modalities in recommendation systems.

Findings

Multi-modal user interactions reveal important cross-modal relationships.

A frequently occurring modality can improve learning from less frequent ones.

The proposed approach effectively handles incomplete multi-modal data.

Abstract

While recommender systems with multi-modal item representations (image, audio, and text), have been widely explored, learning recommendations from multi-modal user interactions (e.g., clicks and speech) remains an open problem. We study the case of multi-modal user interactions in a setting where users engage with a service provider through multiple channels (website and call center). In such cases, incomplete modalities naturally occur, since not all users interact through all the available channels. To address these challenges, we publish a real-world dataset that allows progress in this under-researched area. We further present and benchmark various methods for leveraging multi-modal user interactions for item recommendations, and propose a novel approach that specifically deals with missing modalities by mapping user interactions to a common feature space. Our analysis reveals important interactions between the different modalities and that a frequently occurring modality can enhance learning from a less frequent one.