Generalizing Sports Feedback Generation by Watching Competitions and Reading Books: A Rock Climbing Case Study

TL;DR

This paper introduces a method to improve sports feedback generation by leveraging web data like videos and manuals, specifically for rock climbing, and proposes new evaluation metrics to better assess feedback quality.

Contribution

It presents a novel approach that uses auxiliary web data to enhance feedback generation for unseen sports and introduces metrics for more meaningful evaluation.

Findings

Improved feedback quality on unseen sports using web data.

Proposed metrics better capture feedback specificity and actionability.

Enhanced generalization of feedback models with limited annotations.

Abstract

While there is rapid progress in video-LLMs with advanced reasoning capabilities, prior work shows that these models struggle on the challenging task of sports feedback generation and require expensive and difficult-to-collect finetuning feedback data for each sport. This limitation is evident from the poor generalization to sports unseen during finetuning. Furthermore, traditional text generation evaluation metrics (e.g., BLEU-4, METEOR, ROUGE-L, BERTScore), originally developed for machine translation and summarization, fail to capture the unique aspects of sports feedback quality. To address the first problem, using rock climbing as our case study, we propose using auxiliary freely-available web data from the target domain, such as competition videos and coaching manuals, in addition to existing sports feedback from a disjoint, source domain to improve sports feedback generation performance on the target domain. To improve evaluation, we propose two evaluation metrics: (1) specificity and (2) actionability. Together, our approach enables more meaningful and practical generation of sports feedback under limited annotations.