Efficient Temporal Action Segmentation via Boundary-aware Query Voting

TL;DR

BaFormer introduces a boundary-aware Transformer for efficient temporal action segmentation, achieving comparable or better accuracy with only 6% of the computational cost of previous state-of-the-art methods.

Contribution

The paper proposes BaFormer, a novel boundary-aware Transformer that simplifies and accelerates temporal action segmentation through a single-stage, boundary-aware approach with a voting strategy.

Findings

BaFormer reduces computational time to 6% of DiffAct.

It achieves comparable or better accuracy on benchmark datasets.

The method effectively balances efficiency and performance.

Abstract

Although the performance of Temporal Action Segmentation (TAS) has improved in recent years, achieving promising results often comes with a high computational cost due to dense inputs, complex model structures, and resource-intensive post-processing requirements. To improve the efficiency while keeping the performance, we present a novel perspective centered on per-segment classification. By harnessing the capabilities of Transformers, we tokenize each video segment as an instance token, endowed with intrinsic instance segmentation. To realize efficient action segmentation, we introduce BaFormer, a boundary-aware Transformer network. It employs instance queries for instance segmentation and a global query for class-agnostic boundary prediction, yielding continuous segment proposals. During inference, BaFormer employs a simple yet effective voting strategy to classify boundary-wise segments based on instance segmentation. Remarkably, as a single-stage approach, BaFormer significantly reduces the computational costs, utilizing only 6% of the running time compared to state-of-the-art method DiffAct, while producing better or comparable accuracy over several popular benchmarks. The code for this project is publicly available at https://github.com/peiyao-w/BaFormer.