Continual Action Assessment via Task-Consistent Score-Discriminative Feature Distribution Modeling

TL;DR

This paper introduces a continual learning framework for Action Quality Assessment that sequentially learns new tasks without forgetting, using a novel feature distribution modeling approach with rehearsal and graph-based knowledge decoupling.

Contribution

The paper proposes a unified model for Continual-AQA that maintains task-consistent features using a new rehearsal method and an action general-specific graph to prevent forgetting across sequential tasks.

Findings

The proposed method effectively mitigates forgetting in Continual-AQA.

Experimental results outperform existing continual learning approaches.

The approach demonstrates high versatility across different AQA tasks.

Abstract

Action Quality Assessment (AQA) is a task that tries to answer how well an action is carried out. While remarkable progress has been achieved, existing works on AQA assume that all the training data are visible for training at one time, but do not enable continual learning on assessing new technical actions. In this work, we address such a Continual Learning problem in AQA (Continual-AQA), which urges a unified model to learn AQA tasks sequentially without forgetting. Our idea for modeling Continual-AQA is to sequentially learn a task-consistent score-discriminative feature distribution, in which the latent features express a strong correlation with the score labels regardless of the task or action types.From this perspective, we aim to mitigate the forgetting in Continual-AQA from two aspects. Firstly, to fuse the features of new and previous data into a score-discriminative distribution, a novel Feature-Score Correlation-Aware Rehearsal is proposed to store and reuse data from previous tasks with limited memory size. Secondly, an Action General-Specific Graph is developed to learn and decouple the action-general and action-specific knowledge so that the task-consistent score-discriminative features can be better extracted across various tasks. Extensive experiments are conducted to evaluate the contributions of proposed components. The comparisons with the existing continual learning methods additionally verify the effectiveness and versatility of our approach. Data and code are available at https://github.com/iSEE-Laboratory/Continual-AQA.