SV3.3B: A Sports Video Understanding Model for Action Recognition

TL;DR

SV3.3B is a lightweight, efficient sports video understanding model that combines novel sampling and self-supervised learning to accurately recognize and describe athletic actions with high detail and precision.

Contribution

The paper introduces SV3.3B, a novel 3.3B parameter model that integrates temporal motion sampling and self-supervised learning for on-device sports video analysis.

Findings

Outperforms larger models like GPT-4o in sports description accuracy

Achieves 29.2% improvement in validation metrics over GPT-4o

Demonstrates high information density and action complexity recognition

Abstract

This paper addresses the challenge of automated sports video analysis, which has traditionally been limited by computationally intensive models requiring server-side processing and lacking fine-grained understanding of athletic movements. Current approaches struggle to capture the nuanced biomechanical transitions essential for meaningful sports analysis, often missing critical phases like preparation, execution, and follow-through that occur within seconds. To address these limitations, we introduce SV3.3B, a lightweight 3.3B parameter video understanding model that combines novel temporal motion difference sampling with self-supervised learning for efficient on-device deployment. Our approach employs a DWT-VGG16-LDA based keyframe extraction mechanism that intelligently identifies the 16 most representative frames from sports sequences, followed by a V-DWT-JEPA2 encoder pretrained through mask-denoising objectives and an LLM decoder fine-tuned for sports action description generation. Evaluated on a subset of the NSVA basketball dataset, SV3.3B achieves superior performance across both traditional text generation metrics and sports-specific evaluation criteria, outperforming larger closed-source models including GPT-4o variants while maintaining significantly lower computational requirements. Our model demonstrates exceptional capability in generating technically detailed and analytically rich sports descriptions, achieving 29.2% improvement over GPT-4o in ground truth validation metrics, with substantial improvements in information density, action complexity, and measurement precision metrics essential for comprehensive athletic analysis. Model Available at https://huggingface.co/sportsvision/SV3.3B.