Anticipating Object State Changes in Long Procedural Videos

TL;DR

This paper introduces a new problem of anticipating object state changes in procedural videos, provides curated annotation data, and proposes a novel framework that integrates visual and language features to predict future object states.

Contribution

It presents the first method for object state change anticipation, extending the Ego4D dataset with new annotations and demonstrating the effectiveness of combined visual and NLP cues.

Findings

Proposed method accurately predicts future object state changes.

Integration of visual and language features improves prediction performance.

New annotated dataset (Ego4D-OSCA) supports future research.

Abstract

In this work, we introduce (a) the new problem of anticipating object state changes in images and videos during procedural activities, (b) new curated annotation data for object state change classification based on the Ego4D dataset, and (c) the first method for addressing this challenging problem. Solutions to this new task have important implications in vision-based scene understanding, automated monitoring systems, and action planning. The proposed novel framework predicts object state changes that will occur in the near future due to yet unseen human actions by integrating learned visual features that represent recent visual information with natural language (NLP) features that represent past object state changes and actions. Leveraging the extensive and challenging Ego4D dataset which provides a large-scale collection of first-person perspective videos across numerous interaction scenarios, we introduce an extension noted Ego4D-OSCA that provides new curated annotation data for the object state change anticipation task (OSCA). An extensive experimental evaluation is presented demonstrating the proposed method's efficacy in predicting object state changes in dynamic scenarios. The performance of the proposed approach also underscores the potential of integrating video and linguistic cues to enhance the predictive performance of video understanding systems and lays the groundwork for future research on the new task of object state change anticipation. The source code and the new annotation data (Ego4D-OSCA) will be made publicly available.