Faster Diffusion Action Segmentation

TL;DR

EffiDiffAct is a novel, efficient diffusion-based approach for temporal action segmentation that reduces computational costs and improves accuracy by using lightweight encoders and adaptive strategies.

Contribution

The paper introduces EffiDiffAct, combining a lightweight encoder and adaptive skip strategy to enhance diffusion-based TAS with lower computational demands.

Findings

Outperforms existing methods on 50Salads, Breakfast, and GTEA datasets.

Reduces computational overhead significantly.

Improves segmentation accuracy and efficiency.

Abstract

Temporal Action Segmentation (TAS) is an essential task in video analysis, aiming to segment and classify continuous frames into distinct action segments. However, the ambiguous boundaries between actions pose a significant challenge for high-precision segmentation. Recent advances in diffusion models have demonstrated substantial success in TAS tasks due to their stable training process and high-quality generation capabilities. However, the heavy sampling steps required by diffusion models pose a substantial computational burden, limiting their practicality in real-time applications. Additionally, most related works utilize Transformer-based encoder architectures. Although these architectures excel at capturing long-range dependencies, they incur high computational costs and face feature-smoothing issues when processing long video sequences. To address these challenges, we propose EffiDiffAct, an efficient and high-performance TAS algorithm. Specifically, we develop a lightweight temporal feature encoder that reduces computational overhead and mitigates the rank collapse phenomenon associated with traditional self-attention mechanisms. Furthermore, we introduce an adaptive skip strategy that allows for dynamic adjustment of timestep lengths based on computed similarity metrics during inference, thereby further enhancing computational efficiency. Comprehensive experiments on the 50Salads, Breakfast, and GTEA datasets demonstrated the effectiveness of the proposed algorithm.