Streaming Video Temporal Action Segmentation In Real Time
Wujun Wen, Yunheng Li, Zhuben Dong, Lin Feng, Wanxiao Yang, Shenlan, Liu

TL;DR
This paper introduces a novel multi-modality model for real-time streaming video action segmentation, achieving high accuracy with significantly reduced computation, enabling practical real-time applications.
Contribution
It presents the first multi-modality real-time temporal action segmentation model that operates without future information, combining language and image features for improved performance.
Findings
Achieves 90% accuracy of full video models in real time.
Uses less than 40% of the computation of state-of-the-art models.
First to propose a multi-modality approach for real-time TAS.
Abstract
Temporal action segmentation (TAS) is a critical step toward long-term video understanding. Recent studies follow a pattern that builds models based on features instead of raw video picture information. However, we claim those models are trained complicatedly and limit application scenarios. It is hard for them to segment human actions of video in real time because they must work after the full video features are extracted. As the real-time action segmentation task is different from TAS task, we define it as streaming video real-time temporal action segmentation (SVTAS) task. In this paper, we propose a real-time end-to-end multi-modality model for SVTAS task. More specifically, under the circumstances that we cannot get any future information, we segment the current human action of streaming video chunk in real time. Furthermore, the model we propose combines the last steaming video…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsHuman Pose and Action Recognition · Anomaly Detection Techniques and Applications · Multimodal Machine Learning Applications
