Sports Analysis and VR Viewing System Based on Player Tracking and Pose Estimation with Multimodal and Multiview Sensors

TL;DR

This paper introduces a comprehensive VR/AR sports analysis system utilizing multimodal sensors for real-time player tracking, pose estimation, and immersive visualization, enhancing sports viewing experiences and providing valuable insights.

Contribution

It presents a novel integrated system combining multimodal data collection, limited supervised learning for pose estimation, and real-time VR/AR visualization for sports analysis.

Findings

High accuracy and robustness in multi-player tracking and pose estimation.

Effective real-time visualization demonstrating immersive sports viewing.

Collection of a new multimodal sports dataset and open-source code release.

Abstract

Sports analysis and viewing play a pivotal role in the current sports domain, offering significant value not only to coaches and athletes but also to fans and the media. In recent years, the rapid development of virtual reality (VR) and augmented reality (AR) technologies have introduced a new platform for watching games. Visualization of sports competitions in VR/AR represents a revolutionary technology, providing audiences with a novel immersive viewing experience. However, there is still a lack of related research in this area. In this work, we present for the first time a comprehensive system for sports competition analysis and real-time visualization on VR/AR platforms. First, we utilize multiview LiDARs and cameras to collect multimodal game data. Subsequently, we propose a framework for multi-player tracking and pose estimation based on a limited amount of supervised data, which extracts precise player positions and movements from point clouds and images. Moreover, we perform avatar modeling of players to obtain their 3D models. Ultimately, using these 3D player data, we conduct competition analysis and real-time visualization on VR/AR. Extensive quantitative experiments demonstrate the accuracy and robustness of our multi-player tracking and pose estimation framework. The visualization results showcase the immense potential of our sports visualization system on the domain of watching games on VR/AR devices. The multimodal competition dataset we collected and all related code will be released soon.