Action Quality Assessment with Temporal Parsing Transformer

TL;DR

This paper introduces a temporal parsing transformer for action quality assessment that decomposes videos into fine-grained temporal parts, improving accuracy over holistic methods and achieving state-of-the-art results.

Contribution

The paper proposes a novel temporal parsing transformer with learnable queries and new loss functions to better capture intra-class variations in action quality assessment.

Findings

Outperforms prior methods on three AQA benchmarks

Uses contrastive regression with part-level representations

Introduces ranking and sparsity loss functions for better temporal parsing

Abstract

Action Quality Assessment(AQA) is important for action understanding and resolving the task poses unique challenges due to subtle visual differences. Existing state-of-the-art methods typically rely on the holistic video representations for score regression or ranking, which limits the generalization to capture fine-grained intra-class variation. To overcome the above limitation, we propose a temporal parsing transformer to decompose the holistic feature into temporal part-level representations. Specifically, we utilize a set of learnable queries to represent the atomic temporal patterns for a specific action. Our decoding process converts the frame representations to a fixed number of temporally ordered part representations. To obtain the quality score, we adopt the state-of-the-art contrastive regression based on the part representations. Since existing AQA datasets do not provide temporal part-level labels or partitions, we propose two novel loss functions on the cross attention responses of the decoder: a ranking loss to ensure the learnable queries to satisfy the temporal order in cross attention and a sparsity loss to encourage the part representations to be more discriminative. Extensive experiments show that our proposed method outperforms prior work on three public AQA benchmarks by a considerable margin.