TricorNet: A Hybrid Temporal Convolutional and Recurrent Network for Video Action Segmentation

TL;DR

TricorNet is a hybrid neural network combining temporal convolutional and recurrent layers, designed for improved video action segmentation by capturing local motion and long-term dependencies, outperforming existing methods.

Contribution

The paper introduces TricorNet, a novel hybrid encoder-decoder architecture that effectively models local motion and long-term action dependencies in video segmentation.

Findings

Achieves superior performance on three public datasets.

Effectively captures local motion changes and long-term dependencies.

Outperforms state-of-the-art methods in action segmentation.

Abstract

Action segmentation as a milestone towards building automatic systems to understand untrimmed videos has received considerable attention in the recent years. It is typically being modeled as a sequence labeling problem but contains intrinsic and sufficient differences than text parsing or speech processing. In this paper, we introduce a novel hybrid temporal convolutional and recurrent network (TricorNet), which has an encoder-decoder architecture: the encoder consists of a hierarchy of temporal convolutional kernels that capture the local motion changes of different actions; the decoder is a hierarchy of recurrent neural networks that are able to learn and memorize long-term action dependencies after the encoding stage. Our model is simple but extremely effective in terms of video sequence labeling. The experimental results on three public action segmentation datasets have shown that the proposed model achieves superior performance over the state of the art.