TennisExpert: Towards Expert-Level Analytical Sports Video Understanding

TL;DR

TennisExpert introduces a multimodal framework and a large-scale annotated dataset for expert-level understanding of tennis videos, enabling detailed tactical analysis and real-time commentary.

Contribution

The paper presents TennisVL, a comprehensive tennis benchmark dataset, and TennisExpert, a novel multimodal system that captures tactical reasoning and match dynamics.

Findings

Outperforms proprietary baselines like GPT-5, Gemini, and Claude

Effectively captures tactical context and match momentum

Provides publicly available dataset and code for further research

Abstract

Tennis is one of the most widely followed sports, generating extensive broadcast footage with strong potential for professional analysis, automated coaching, and real-time commentary. However, automatic tennis understanding remains underexplored due to two key challenges: (1) the lack of large-scale benchmarks with fine-grained annotations and expert-level commentary, and (2) the difficulty of building accurate yet efficient multimodal systems suitable for real-time deployment. To address these challenges, we introduce TennisVL, a large-scale tennis benchmark comprising over 200 professional matches (471.9 hours) and 40,000+ rally-level clips. Unlike existing commentary datasets that focus on descriptive play-by-play narration, TennisVL emphasizes expert analytical commentary capturing tactical reasoning, player decisions, and match momentum. Furthermore, we propose TennisExpert, a multimodal tennis understanding framework that integrates a video semantic parser with a memory-augmented model built on Qwen3-VL-8B. The parser extracts key match elements (e.g., scores, shot sequences, ball bounces, and player locations), while hierarchical memory modules capture both short- and long-term temporal context. Experiments show that TennisExpert consistently outperforms strong proprietary baselines, including GPT-5, Gemini, and Claude, and demonstrates improved ability to capture tactical context and match dynamics. Our dataset and code are publicly available at https://github.com/LZYAndy/TennisExpert.