Improving Temporal Action Segmentation via Constraint-Aware Decoding

TL;DR

This paper introduces a lightweight, constraint-based refinement method for temporal action segmentation that improves prediction accuracy by integrating structural priors into a modified Viterbi decoding process.

Contribution

It presents a novel, efficient framework that enhances TAS predictions using statistical structural priors without retraining or increasing model complexity.

Findings

Improves both fully and semi-supervised TAS models.

Corrects structural prediction errors effectively.

Maintains high inference efficiency.

Abstract

Temporal action segmentation (TAS) divides untrimmed videos into labeled action segments. While fully supervised methods have advanced the field, challenges such as action variability, ambiguous boundaries, and high annotation costs remain, especially in new or low-resource domains. Grammar-based approaches improve segmentation with structural priors but rely on complex parsing limiting scalability. In this work, we propose a lightweight, constraint-based refinement framework that enhances TAS predictions by integrating statistical structural priors such as transition confidence, action boundary sets, and per-class duration, that can be directly extracted from annotated data. These constraints are integrated into a modified Viterbi decoding algorithm, allowing inference-time refinement without retraining or added model complexity. Our approach improves both fully and semi-supervised TAS models by correcting structural prediction errors while maintaining high efficiency. Code is available at https://github.com/LUNAProject22/CAD