HopaDIFF: Holistic-Partial Aware Fourier Conditioned Diffusion for Referring Human Action Segmentation in Multi-Person Scenarios

TL;DR

This paper introduces HopaDIFF, a novel diffusion-based framework for referring human action segmentation in multi-person videos, along with a new dataset RHAS133, addressing the limitations of existing single-person focused methods.

Contribution

The work pioneers multi-person action segmentation guided by textual references, introduces a new dataset RHAS133, and proposes a Fourier-conditioned diffusion model with a cross-input gate for improved reasoning.

Findings

HopaDIFF outperforms existing methods on RHAS133

The new dataset RHAS133 contains 137 fine-grained actions in 133 movies

Benchmarking shows limited performance of existing methods on multi-person scenarios

Abstract

Action segmentation is a core challenge in high-level video understanding, aiming to partition untrimmed videos into segments and assign each a label from a predefined action set. Existing methods primarily address single-person activities with fixed action sequences, overlooking multi-person scenarios. In this work, we pioneer textual reference-guided human action segmentation in multi-person settings, where a textual description specifies the target person for segmentation. We introduce the first dataset for Referring Human Action Segmentation, i.e., RHAS133, built from 133 movies and annotated with 137 fine-grained actions with 33h video data, together with textual descriptions for this new task. Benchmarking existing action segmentation methods on RHAS133 using VLM-based feature extractors reveals limited performance and poor aggregation of visual cues for the target person. To address this, we propose a holistic-partial aware Fourier-conditioned diffusion framework, i.e., HopaDIFF, leveraging a novel cross-input gate attentional xLSTM to enhance holistic-partial long-range reasoning and a novel Fourier condition to introduce more fine-grained control to improve the action segmentation generation. HopaDIFF achieves state-of-the-art results on RHAS133 in diverse evaluation settings. The dataset and code are available at https://github.com/KPeng9510/HopaDIFF.