Enhancing Multimodal Affective Analysis with Learned Live Comment Features

TL;DR

This paper introduces a new dataset and a contrastive learning approach to generate synthetic live comment features, significantly improving multimodal affective analysis across multiple tasks and languages.

Contribution

The work creates the LCAffect dataset and develops a contrastive learning method to enhance affective analysis with synthetic live comment features.

Findings

Synthetic live comment features improve affective analysis performance.

The approach is effective across multiple languages and tasks.

The dataset enables broader research in live comment-based emotion detection.

Abstract

Live comments, also known as Danmaku, are user-generated messages that are synchronized with video content. These comments overlay directly onto streaming videos, capturing viewer emotions and reactions in real-time. While prior work has leveraged live comments in affective analysis, its use has been limited due to the relative rarity of live comments across different video platforms. To address this, we first construct the Live Comment for Affective Analysis (LCAffect) dataset which contains live comments for English and Chinese videos spanning diverse genres that elicit a wide spectrum of emotions. Then, using this dataset, we use contrastive learning to train a video encoder to produce synthetic live comment features for enhanced multimodal affective content analysis. Through comprehensive experimentation on a wide range of affective analysis tasks (sentiment, emotion recognition, and sarcasm detection) in both English and Chinese, we demonstrate that these synthetic live comment features significantly improve performance over state-of-the-art methods.