Impact of Stickers on Multimodal Sentiment and Intent in Social Media: A New Task, Dataset and Baseline

TL;DR

This paper introduces a new multimodal task and dataset for analyzing the impact of stickers on sentiment and intent in social media chats, along with a baseline model that outperforms existing approaches.

Contribution

It presents a novel task, a comprehensive dataset with Chinese chat records and stickers, and a multimodal model that improves sentiment and intent recognition accuracy.

Findings

Joint modeling of sentiment and intent improves accuracy.

The MMSAIR model outperforms traditional and advanced models.

Stickers significantly influence chat sentiment and intent understanding.

Abstract

Stickers are increasingly used in social media to express sentiment and intent. Despite their significant impact on sentiment analysis and intent recognition, little research has been conducted in this area. To address this gap, we propose a new task: \textbf{M}ultimodal chat \textbf{S}entiment \textbf{A}nalysis and \textbf{I}ntent \textbf{R}ecognition involving \textbf{S}tickers (MSAIRS). Additionally, we introduce a novel multimodal dataset containing Chinese chat records and stickers excerpted from several mainstream social media platforms. Our dataset includes paired data with the same text but different stickers, the same sticker but different contexts, and various stickers consisting of the same images with different texts, allowing us to better understand the impact of stickers on chat sentiment and intent. We also propose an effective multimodal joint model, MMSAIR, featuring differential vector construction and cascaded attention mechanisms for enhanced multimodal fusion. Our experiments demonstrate the necessity and effectiveness of jointly modeling sentiment and intent, as they mutually reinforce each other's recognition accuracy. MMSAIR significantly outperforms traditional models and advanced MLLMs, demonstrating the challenge and uniqueness of sticker interpretation in social media. Our dataset and code are available on https://github.com/FakerBoom/MSAIRS-Dataset.