CaFlow: Enhancing Long-Term Action Quality Assessment with Causal Counterfactual Flow

TL;DR

CaFlow introduces a novel framework combining causal de-confounding and bidirectional flow modeling to improve long-term action quality assessment, achieving state-of-the-art results in challenging video analysis tasks.

Contribution

The paper presents CaFlow, a unified approach that integrates counterfactual de-confounding with bidirectional flow to enhance robustness and accuracy in long-term action quality assessment.

Findings

Achieves state-of-the-art performance on multiple benchmarks.

Effectively disentangles causal and confounding features.

Produces smoother, more coherent representations of long-term actions.

Abstract

Action Quality Assessment (AQA) predicts fine-grained execution scores from action videos and is widely applied in sports, rehabilitation, and skill evaluation. Long-term AQA, as in figure skating or rhythmic gymnastics, is especially challenging since it requires modeling extended temporal dynamics while remaining robust to contextual confounders. Existing approaches either depend on costly annotations or rely on unidirectional temporal modeling, making them vulnerable to spurious correlations and unstable long-term representations. To this end, we propose CaFlow, a unified framework that integrates counterfactual de-confounding with bidirectional time-conditioned flow. The Causal Counterfactual Regularization (CCR) module disentangles causal and confounding features in a self-supervised manner and enforces causal robustness through counterfactual interventions, while the BiT-Flow module models forward and backward dynamics with a cycle-consistency constraint to produce smoother and more coherent representations. Extensive experiments on multiple long-term AQA benchmarks demonstrate that CaFlow achieves state-of-the-art performance. Code is available at https://github.com/Harrison21/CaFlow