Do We Trust What They Say or What They Do? A Multimodal User Embedding Provides Personalized Explanations

TL;DR

This paper introduces CAMUE, a multimodal user embedding framework that combines text and graph data to produce personalized, explainable predictions in social networks, enhancing trustworthiness and content delivery.

Contribution

The paper presents a novel contribution-aware multimodal embedding method that improves explainability and reliability in social media user analysis.

Findings

Graph structure information is generally more trustworthy than text.

Text information can be more helpful for certain users.

The approach mitigates the impact of unreliable information.

Abstract

With the rapid development of social media, the importance of analyzing social network user data has also been put on the agenda. User representation learning in social media is a critical area of research, based on which we can conduct personalized content delivery, or detect malicious actors. Being more complicated than many other types of data, social network user data has inherent multimodal nature. Various multimodal approaches have been proposed to harness both text (i.e. post content) and relation (i.e. inter-user interaction) information to learn user embeddings of higher quality. The advent of Graph Neural Network models enables more end-to-end integration of user text embeddings and user interaction graphs in social networks. However, most of those approaches do not adequately elucidate which aspects of the data - text or graph structure information - are more helpful for predicting each specific user under a particular task, putting some burden on personalized downstream analysis and untrustworthy information filtering. We propose a simple yet effective framework called Contribution-Aware Multimodal User Embedding (CAMUE) for social networks. We have demonstrated with empirical evidence, that our approach can provide personalized explainable predictions, automatically mitigating the impact of unreliable information. We also conducted case studies to show how reasonable our results are. We observe that for most users, graph structure information is more trustworthy than text information, but there are some reasonable cases where text helps more. Our work paves the way for more explainable, reliable, and effective social media user embedding which allows for better personalized content delivery.