ASFormer: Transformer for Action Segmentation

TL;DR

ASFormer is an efficient Transformer-based model designed specifically for action segmentation, incorporating local priors, hierarchical input handling, and a refined decoder to improve accuracy on long sequences with limited training data.

Contribution

The paper introduces ASFormer, a novel Transformer architecture tailored for action segmentation, addressing issues of local feature modeling, long sequence processing, and prediction refinement.

Findings

Outperforms existing methods on three public datasets.

Effectively models long input sequences with hierarchical design.

Improves segmentation accuracy with small training sets.

Abstract

Algorithms for the action segmentation task typically use temporal models to predict what action is occurring at each frame for a minute-long daily activity. Recent studies have shown the potential of Transformer in modeling the relations among elements in sequential data. However, there are several major concerns when directly applying the Transformer to the action segmentation task, such as the lack of inductive biases with small training sets, the deficit in processing long input sequence, and the limitation of the decoder architecture to utilize temporal relations among multiple action segments to refine the initial predictions. To address these concerns, we design an efficient Transformer-based model for action segmentation task, named ASFormer, with three distinctive characteristics: (i) We explicitly bring in the local connectivity inductive priors because of the high locality of features. It constrains the hypothesis space within a reliable scope, and is beneficial for the action segmentation task to learn a proper target function with small training sets. (ii) We apply a pre-defined hierarchical representation pattern that efficiently handles long input sequences. (iii) We carefully design the decoder to refine the initial predictions from the encoder. Extensive experiments on three public datasets demonstrate that effectiveness of our methods. Code is available at \url{https://github.com/ChinaYi/ASFormer}.