Learning Consistent Temporal Grounding between Related Tasks in Sports Coaching

TL;DR

This paper introduces a method to improve temporal grounding in sports coaching videos by enforcing self-consistency between related tasks' attention maps, leading to significant accuracy gains without extra annotations.

Contribution

It proposes a novel self-consistency training approach that leverages related tasks to enhance temporal grounding in video language models without additional supervision.

Findings

Attention misallocation is a key bottleneck in temporal grounding.

Self-consistency training improves accuracy by up to 14.1%.

Method surpasses some closed-source models in sports coaching tasks.

Abstract

Video-LLMs often attend to irrelevant frames, which is especially detrimental for sports coaching tasks requiring precise temporal grounding. Yet obtaining frame-level supervision is challenging: expensive to collect from humans and unreliable from other models. We improve temporal grounding without additional annotations by exploiting the observation that related tasks, such as generation and verification, must attend to the same frames. We enforce this via a self-consistency objective over select visual attention maps of tightly-related tasks. Using VidDiffBench, which provides ground-truth keyframe annotations, we first validate that attention misallocation is a significant bottleneck. We then show that training with our objective yields gains of +3.0%, +14.1% accuracy and +0.9 BERTScore over supervised finetuning across three sports coaching tasks: Exact, FitnessQA, and ExpertAF, even surpassing closed-source models.