TechCoach: Towards Technical-Point-Aware Descriptive Action Coaching

TL;DR

This paper introduces TechCoach, a framework for detailed, point-aware action coaching that provides comprehensive feedback on action quality at the TechPoint level, supported by a new dataset and benchmark.

Contribution

It presents a novel dataset EE4D-DescCoach with detailed TechPoint annotations and a framework that explicitly reasons about TechPoints for improved descriptive coaching.

Findings

TechCoach outperforms baseline models in providing detailed coaching comments.

The dataset enables fine-grained action assessment at the TechPoint level.

Explicit TechPoint reasoning improves the quality of coaching feedback.

Abstract

To guide a learner in mastering action skills, it is crucial for a coach to 1) reason through the learner's action execution and technical points (TechPoints), and 2) provide detailed, comprehensible feedback on what is done well and what can be improved. However, existing score-based action assessment methods are still far from reaching this practical scenario. To bridge this gap, we investigate a new task termed Descriptive Action Coaching (DescCoach) which requires the model to provide detailed commentary on what is done well and what can be improved beyond a simple quality score for action execution. To this end, we first build a new dataset named EE4D-DescCoach. Through an automatic annotation pipeline, our dataset goes beyond the existing action assessment datasets by providing detailed TechPoint-level commentary. Furthermore, we propose TechCoach, a new framework that explicitly incorporates TechPoint-level reasoning into the DescCoach process. The central to our method lies in the Context-aware TechPoint Reasoner, which enables TechCoach to learn TechPoint-related quality representation by querying visual context under the supervision of TechPoint-level coaching commentary. By leveraging the visual context and the TechPoint-related quality representation, a unified TechPoint-aware Action Assessor is then employed to provide the overall coaching commentary together with the quality score. Combining all of these, we establish a new benchmark for DescCoach and evaluate the effectiveness of our method through extensive experiments. The data and code will be made publicly available.