Auto-Encoding Score Distribution Regression for Action Quality Assessment

TL;DR

This paper introduces a novel distribution auto-encoder module to model uncertainty in action quality assessment, significantly improving accuracy on multiple datasets by better capturing the video-score relationship.

Contribution

It develops a plug-and-play distribution auto-encoder for AQA that models aleatoric uncertainty, enhancing existing methods with a probabilistic approach.

Findings

Achieves state-of-the-art results on AQA-7, MTL-AQA, and JIGSAWS datasets.

Effectively models uncertainty in video-score mappings.

Improves regression accuracy in action quality assessment.

Abstract

The action quality assessment (AQA) of videos is a challenging vision task since the relation between videos and action scores is difficult to model. Thus, AQA has been widely studied in the literature. Traditionally, AQA is treated as a regression problem to learn the underlying mappings between videos and action scores. But previous methods ignored data uncertainty in AQA dataset. To address aleatoric uncertainty, we further develop a plug-and-play module Distribution Auto-Encoder (DAE). Specifically, it encodes videos into distributions and uses the reparameterization trick in variational auto-encoders (VAE) to sample scores, which establishes a more accurate mapping between videos and scores. Meanwhile, a likelihood loss is used to learn the uncertainty parameters. We plug our DAE approach into MUSDL and CoRe. Experimental results on public datasets demonstrate that our method achieves state-of-the-art on AQA-7, MTL-AQA, and JIGSAWS datasets. Our code is available at https://github.com/InfoX-SEU/DAE-AQA.