StoryNavi: On-Demand Narrative-Driven Reconstruction of Video Play With Generative AI

TL;DR

StoryNavi leverages generative AI and large language models to create customized, narrative-driven, non-linear video plays from lengthy videos, enhancing user engagement and retrieval efficiency.

Contribution

The paper introduces a novel system that constructs personalized, narrative-based video sequences from original videos using retrieval and generative AI, enabling flexible playback modes.

Findings

Adequate retrieval performance compared to human retrieval

Narrative coherence improves user engagement

User preferences vary based on video genre and query

Abstract

Manually navigating lengthy videos to seek information or answer questions can be a tedious and time-consuming task for users. We introduce StoryNavi, a novel system powered by VLLMs for generating customised video play experiences by retrieving materials from original videos. It directly answers users' query by constructing non-linear sequence with identified relevant clips to form a cohesive narrative. StoryNavi offers two modes of playback of the constructed video plays: 1) video-centric, which plays original audio and skips irrelevant segments, and 2) narrative-centric, narration guides the experience, and the original audio is muted. Our technical evaluation showed adequate retrieval performance compared to human retrieval. Our user evaluation shows that maintaining narrative coherence significantly enhances user engagement when viewing disjointed video segments. However, factors like video genre, content, and the query itself may lead to varying user preferences for the playback mode.