TikTalk: A Video-Based Dialogue Dataset for Multi-Modal Chitchat in Real World

TL;DR

TikTalk is a new large-scale video-based multi-modal dialogue dataset that captures real-world chitchat, presenting unique challenges and opportunities for developing more human-like multi-modal conversational AI.

Contribution

The paper introduces TikTalk, a comprehensive video-based multi-modal dialogue dataset with diverse context types, and evaluates baseline models, highlighting the potential of LLMs and external knowledge integration.

Findings

Models with large language models generate more diverse responses.

Knowledge graph-based models perform best overall.

Current models still struggle with complex multi-modal understanding.

Abstract

To facilitate the research on intelligent and human-like chatbots with multi-modal context, we introduce a new video-based multi-modal dialogue dataset, called TikTalk. We collect 38K videos from a popular video-sharing platform, along with 367K conversations posted by users beneath them. Users engage in spontaneous conversations based on their multi-modal experiences from watching videos, which helps recreate real-world chitchat context. Compared to previous multi-modal dialogue datasets, the richer context types in TikTalk lead to more diverse conversations, but also increase the difficulty in capturing human interests from intricate multi-modal information to generate personalized responses. Moreover, external knowledge is more frequently evoked in our dataset. These facts reveal new challenges for multi-modal dialogue models. We quantitatively demonstrate the characteristics of TikTalk, propose a video-based multi-modal chitchat task, and evaluate several dialogue baselines. Experimental results indicate that the models incorporating large language models (LLM) can generate more diverse responses, while the model utilizing knowledge graphs to introduce external knowledge performs the best overall. Furthermore, no existing model can solve all the above challenges well. There is still a large room for future improvements, even for LLM with visual extensions. Our dataset is available at \url{https://ruc-aimind.github.io/projects/TikTalk/}.