Large VLM-based Stylized Sports Captioning

TL;DR

This paper introduces a two-level fine-tuned LVLM pipeline that significantly improves the accuracy and style of sports image captions, demonstrating practical real-time application in live sports journalism.

Contribution

It presents a novel two-level fine-tuning approach for LVLMs to generate accurate, stylized sports captions, addressing limitations of existing models.

Findings

8-10% improvement in F1 score

2-10% enhancement in BERT score

Real-time captioning at 6 images per 3-5 seconds

Abstract

The advent of large (visual) language models (LLM / LVLM) have led to a deluge of automated human-like systems in several domains including social media content generation, search and recommendation, healthcare prognosis, AI assistants for cognitive tasks etc. Although these systems have been successfully integrated in production; very little focus has been placed on sports, particularly accurate identification and natural language description of the game play. Most existing LLM/LVLMs can explain generic sports activities, but lack sufficient domain-centric sports' jargon to create natural (human-like) descriptions. This work highlights the limitations of existing SoTA LLM/LVLMs for generating production-grade sports captions from images in a desired stylized format, and proposes a two-level fine-tuned LVLM pipeline to address that. The proposed pipeline yields an improvement > 8-10% in the F1, and > 2-10% in BERT score compared to alternative approaches. In addition, it has a small runtime memory footprint and fast execution time. During Super Bowl LIX the pipeline proved its practical application for live professional sports journalism; generating highly accurate and stylized captions at the rate of 6 images per 3-5 seconds for over 1000 images during the game play.