Learning Object State Changes in Videos: An Open-World Perspective

TL;DR

This paper introduces VidOSC, a novel approach leveraging vision-language models for open-world video object state change localization, and presents HowToChange, a new benchmark with extensive annotations, improving generalization beyond closed-world limits.

Contribution

The paper proposes a new open-world formulation for video object state change localization, utilizing text and vision-language models, and introduces the HowToChange benchmark for comprehensive evaluation.

Findings

VidOSC effectively localizes object state changes in open-world scenarios.

The approach outperforms existing methods in both closed and open-world settings.

HowToChange provides a significantly larger and more diverse dataset for evaluation.

Abstract

Object State Changes (OSCs) are pivotal for video understanding. While humans can effortlessly generalize OSC understanding from familiar to unknown objects, current approaches are confined to a closed vocabulary. Addressing this gap, we introduce a novel open-world formulation for the video OSC problem. The goal is to temporally localize the three stages of an OSC -- the object's initial state, its transitioning state, and its end state -- whether or not the object has been observed during training. Towards this end, we develop VidOSC, a holistic learning approach that: (1) leverages text and vision-language models for supervisory signals to obviate manually labeling OSC training data, and (2) abstracts fine-grained shared state representations from objects to enhance generalization. Furthermore, we present HowToChange, the first open-world benchmark for video OSC localization, which offers an order of magnitude increase in the label space and annotation volume compared to the best existing benchmark. Experimental results demonstrate the efficacy of our approach, in both traditional closed-world and open-world scenarios.