TV-Dialogue: Crafting Theme-Aware Video Dialogues with Immersive Interaction

TL;DR

This paper introduces TV-Dialogue, a multi-modal framework for generating theme-aware, immersive video dialogues that align with video content and user-specified themes, enabling real-time character interactions without training.

Contribution

The paper presents a novel multi-modal agent framework for theme-aware video dialogue generation, along with a new evaluation benchmark and zero-shot capabilities.

Findings

Effective dialogue generation for videos of any length and theme

Outperforms existing LLMs in content alignment and visual consistency

Enables applications like video re-creation and film dubbing

Abstract

Recent advancements in LLMs have accelerated the development of dialogue generation across text and images, yet video-based dialogue generation remains underexplored and presents unique challenges. In this paper, we introduce Theme-aware Video Dialogue Crafting (TVDC), a novel task aimed at generating new dialogues that align with video content and adhere to user-specified themes. We propose TV-Dialogue, a novel multi-modal agent framework that ensures both theme alignment (i.e., the dialogue revolves around the theme) and visual consistency (i.e., the dialogue matches the emotions and behaviors of characters in the video) by enabling real-time immersive interactions among video characters, thereby accurately understanding the video content and generating new dialogue that aligns with the given themes. To assess the generated dialogues, we present a multi-granularity evaluation benchmark with high accuracy, interpretability and reliability, demonstrating the effectiveness of TV-Dialogue on self-collected dataset over directly using existing LLMs. Extensive experiments reveal that TV-Dialogue can generate dialogues for videos of any length and any theme in a zero-shot manner without training. Our findings underscore the potential of TV-Dialogue for various applications, such as video re-creation, film dubbing and its use in downstream multimodal tasks.