Towards Open-World Human Action Segmentation Using Graph Convolutional Networks

TL;DR

This paper introduces a novel framework for open-world human action segmentation using graph convolutional networks, capable of detecting and segmenting unseen actions without manual annotations, advancing activity understanding in dynamic environments.

Contribution

The paper proposes an innovative structured framework with a new GCN decoder, Mixup training for out-of-distribution data synthesis, and a Temporal Clustering loss for improved open-world action segmentation.

Findings

Significant improvements in open-set segmentation metrics (F1@50)

Enhanced out-of-distribution detection performance (AUROC)

Effective ablation of proposed components

Abstract

Human-object interaction segmentation is a fundamental task of daily activity understanding, which plays a crucial role in applications such as assistive robotics, healthcare, and autonomous systems. Most existing learning-based methods excel in closed-world action segmentation, they struggle to generalize to open-world scenarios where novel actions emerge. Collecting exhaustive action categories for training is impractical due to the dynamic diversity of human activities, necessitating models that detect and segment out-of-distribution actions without manual annotation. To address this issue, we formally define the open-world action segmentation problem and propose a structured framework for detecting and segmenting unseen actions. Our framework introduces three key innovations: 1) an Enhanced Pyramid Graph Convolutional Network (EPGCN) with a novel decoder module for robust spatiotemporal feature upsampling. 2) Mixup-based training to synthesize out-of-distribution data, eliminating reliance on manual annotations. 3) A novel Temporal Clustering loss that groups in-distribution actions while distancing out-of-distribution samples. We evaluate our framework on two challenging human-object interaction recognition datasets: Bimanual Actions and 2 Hands and Object (H2O) datasets. Experimental results demonstrate significant improvements over state-of-the-art action segmentation models across multiple open-set evaluation metrics, achieving 16.9% and 34.6% relative gains in open-set segmentation (F1@50) and out-of-distribution detection performances (AUROC), respectively. Additionally, we conduct an in-depth ablation study to assess the impact of each proposed component, identifying the optimal framework configuration for open-world action segmentation.