OnlineTAS: An Online Baseline for Temporal Action Segmentation

TL;DR

This paper introduces an online framework for temporal action segmentation that uses adaptive memory and feature augmentation to effectively capture context in real-time, achieving state-of-the-art results on benchmark datasets.

Contribution

It proposes a novel online approach with adaptive memory and feature augmentation for temporal action segmentation, addressing the challenge of real-time context modeling.

Findings

Achieves state-of-the-art performance on three benchmarks.

Effectively mitigates over-segmentation in online setting.

Demonstrates the effectiveness of adaptive memory in real-time segmentation.

Abstract

Temporal context plays a significant role in temporal action segmentation. In an offline setting, the context is typically captured by the segmentation network after observing the entire sequence. However, capturing and using such context information in an online setting remains an under-explored problem. This work presents the an online framework for temporal action segmentation. At the core of the framework is an adaptive memory designed to accommodate dynamic changes in context over time, alongside a feature augmentation module that enhances the frames with the memory. In addition, we propose a post-processing approach to mitigate the severe over-segmentation in the online setting. On three common segmentation benchmarks, our approach achieves state-of-the-art performance.