Sportify: Question Answering with Embedded Visualizations and Personified Narratives for Sports Video

TL;DR

Sportify is a system that combines embedded visualizations and personified narratives, generated by large-language models, to help basketball fans understand complex tactics and improve their viewing experience.

Contribution

It introduces novel action visualizations and a storytelling approach that integrates narratives with visualizations, enhancing tactical understanding for basketball fans.

Findings

Sportify improves fans' understanding of basketball tactics.

Third-person narration provides in-depth game explanations.

First-person narration increases fan engagement.

Abstract

As basketball's popularity surges, fans often find themselves confused and overwhelmed by the rapid game pace and complexity. Basketball tactics, involving a complex series of actions, require substantial knowledge to be fully understood. This complexity leads to a need for additional information and explanation, which can distract fans from the game. To tackle these challenges, we present Sportify, a Visual Question Answering system that integrates narratives and embedded visualization for demystifying basketball tactical questions, aiding fans in understanding various game aspects. We propose three novel action visualizations (i.e., Pass, Cut, and Screen) to demonstrate critical action sequences. To explain the reasoning and logic behind players' actions, we leverage a large-language model (LLM) to generate narratives. We adopt a storytelling approach for complex scenarios from both first and third-person perspectives, integrating action visualizations. We evaluated Sportify with basketball fans to investigate its impact on understanding of tactics, and how different personal perspectives of narratives impact the understanding of complex tactic with action visualizations. Our evaluation with basketball fans demonstrates Sportify's capability to deepen tactical insights and amplify the viewing experience. Furthermore, third-person narration assists people in getting in-depth game explanations while first-person narration enhances fans' game engagement