Learning Skill-Attributes for Transferable Assessment in Video

TL;DR

This paper introduces CrossTrainer, a transferable video skill assessment model that identifies universal skill-attributes and generates actionable feedback, significantly improving cross-sport and intra-sport evaluation accuracy.

Contribution

The paper presents a novel approach to skill assessment that abstracts shared skill-attributes across sports, enabling transferability and better generalization than existing methods.

Findings

Achieves up to 60% relative improvement over state-of-the-art methods.

Effectively transfers skill assessment across different sports.

Generates meaningful, actionable feedback for diverse sports videos.

Abstract

Skill assessment from video entails rating the quality of a person's physical performance and explaining what could be done better. Today's models specialize for an individual sport, and suffer from the high cost and scarcity of expert-level supervision across the long tail of sports. Towards closing that gap, we explore transferable video representations for skill assessment. Our CrossTrainer approach discovers skill-attributes, such as balance, control, and hand positioning -- whose meaning transcends the boundaries of any given sport, then trains a multimodal language model to generate actionable feedback for a novel video, e.g., "lift hands more to generate more power" as well as its proficiency level, e.g., early expert. We validate the new model on multiple datasets for both cross-sport (transfer) and intra-sport (in-domain) settings, where it achieves gains up to 60% relative to the state of the art. By abstracting out the shared behaviors indicative of human skill, the proposed video representation generalizes substantially better than an array of existing techniques, enriching today's multimodal large language models.