Sports Intelligence: Assessing the Sports Understanding Capabilities of Language Models through Question Answering from Text to Video

TL;DR

This paper evaluates the sports understanding capabilities of large language and video models through a new benchmark, highlighting current challenges and proposing directions for future research in multimodal sports reasoning.

Contribution

It introduces a comprehensive sports understanding benchmark for NLP and video models, including error analysis and evaluation strategies from simple to complex reasoning tasks.

Findings

Mainstream models struggle with complex sports reasoning tasks.

Multimodal models show potential but still face significant challenges.

The new benchmark highlights key areas for future research in sports NLP and video understanding.

Abstract

Understanding sports is crucial for the advancement of Natural Language Processing (NLP) due to its intricate and dynamic nature. Reasoning over complex sports scenarios has posed significant challenges to current NLP technologies which require advanced cognitive capabilities. Toward addressing the limitations of existing benchmarks on sports understanding in the NLP field, we extensively evaluated mainstream large language models for various sports tasks. Our evaluation spans from simple queries on basic rules and historical facts to complex, context-specific reasoning, leveraging strategies from zero-shot to few-shot learning, and chain-of-thought techniques. In addition to unimodal analysis, we further assessed the sports reasoning capabilities of mainstream video language models to bridge the gap in multimodal sports understanding benchmarking. Our findings highlighted the critical challenges of sports understanding for NLP. We proposed a new benchmark based on a comprehensive overview of existing sports datasets and provided extensive error analysis which we hope can help identify future research priorities in this field.