Looking into the Unknown: Exploring Action Discovery for Segmentation of Known and Unknown Actions

TL;DR

This paper introduces Action Discovery, a new approach for segmenting both known and unknown actions in partially labeled datasets, using a two-step method to identify and classify unknown actions based on limited annotations.

Contribution

The paper proposes a novel Action Discovery setup and a two-step approach with GGSM and UASA modules to improve segmentation and classification of unknown actions in partially labeled data.

Findings

Significant improvement over baselines on Breakfast, 50Salads, and Desktop Assembly datasets.

Effective identification of unknown action segments and their semantic classification.

Demonstrates the importance of leveraging known annotations to discover unknown actions.

Abstract

We introduce Action Discovery, a novel setup within Temporal Action Segmentation that addresses the challenge of defining and annotating ambiguous actions and incomplete annotations in partially labeled datasets. In this setup, only a subset of actions - referred to as known actions - is annotated in the training data, while other unknown actions remain unlabeled. This scenario is particularly relevant in domains like neuroscience, where well-defined behaviors (e.g., walking, eating) coexist with subtle or infrequent actions that are often overlooked, as well as in applications where datasets are inherently partially annotated due to ambiguous or missing labels. To address this problem, we propose a two-step approach that leverages the known annotations to guide both the temporal and semantic granularity of unknown action segments. First, we introduce the Granularity-Guided Segmentation Module (GGSM), which identifies temporal intervals for both known and unknown actions by mimicking the granularity of annotated actions. Second, we propose the Unknown Action Segment Assignment (UASA), which identifies semantically meaningful classes within the unknown actions, based on learned embedding similarities. We systematically explore the proposed setting of Action Discovery on three challenging datasets - Breakfast, 50Salads, and Desktop Assembly - demonstrating that our method considerably improves upon existing baselines.