LiveChat: Video Comment Generation from Audio-Visual Multimodal Contexts

TL;DR

This paper introduces a large-scale audio-visual dialogue dataset from Twitch and proposes a multimodal model for generating live comments that reflect video content and dialogue context, advancing live video interaction AI.

Contribution

The creation of a diverse, large-scale dataset and a novel multimodal comment generation model tailored for live streaming videos.

Findings

The dataset includes 438 hours of video and 3.2 million comments.

The proposed model effectively generates contextually relevant live comments.

Initial results demonstrate the model's potential for real-time live video interaction.

Abstract

Live commenting on video, a popular feature of live streaming platforms, enables viewers to engage with the content and share their comments, reactions, opinions, or questions with the streamer or other viewers while watching the video or live stream. It presents a challenging testbed for AI agents, which involves the simultaneous understanding of audio-visual multimodal contexts from live streams and the ability to interact with human viewers through dialogue. As existing live streaming-based comments datasets contain limited categories and lack a diversity, we create a large-scale audio-visual multimodal dialogue dataset to facilitate the development of live commenting technologies. The data is collected from Twitch, with 11 different categories and 575 streamers for a total of 438 hours of video and 3.2 million comments. Moreover, we propose a novel multimodal generation model capable of generating live comments that align with the temporal and spatial events within the video, as well as with the ongoing multimodal dialogue context. Our initial results have demonstrated the effectiveness of the proposed model, providing a robust foundation for further research and practical applications in the field of live video interaction.