# Fast Weakly Supervised Action Segmentation Using Mutual Consistency

## TL;DR

This paper introduces a fast, end-to-end weakly supervised action segmentation method using a two-branch neural network and a novel mutual consistency loss, achieving state-of-the-art accuracy with significantly reduced training and inference time.

## Contribution

It proposes a novel mutual consistency loss for weakly supervised action segmentation, enabling faster training and inference while maintaining high accuracy.

## Key findings

- Achieves state-of-the-art accuracy with weak supervision.
- Training is 14 times faster, inference 20 times faster.
- Mutual consistency loss benefits fully supervised settings.

## Abstract

Action segmentation is the task of predicting the actions for each frame of a video. As obtaining the full annotation of videos for action segmentation is expensive, weakly supervised approaches that can learn only from transcripts are appealing. In this paper, we propose a novel end-to-end approach for weakly supervised action segmentation based on a two-branch neural network. The two branches of our network predict two redundant but different representations for action segmentation and we propose a novel mutual consistency (MuCon) loss that enforces the consistency of the two redundant representations. Using the MuCon loss together with a loss for transcript prediction, our proposed approach achieves the accuracy of state-of-the-art approaches while being $14$ times faster to train and $20$ times faster during inference. The MuCon loss proves beneficial even in the fully supervised setting.