Tracking and Understanding Object Transformations

TL;DR

This paper introduces the task of tracking objects through transformations, proposing a new system called TubeletGraph that improves tracking accuracy and understanding of object state changes, supported by a new benchmark dataset.

Contribution

The paper presents TubeletGraph, a zero-shot system for tracking objects through transformations, and introduces VOST-TAS, a benchmark dataset for this task.

Findings

TubeletGraph achieves state-of-the-art performance in tracking through transformations.

The system demonstrates deep understanding of object state changes.

It shows promising capabilities in temporal grounding and semantic reasoning.

Abstract

Real-world objects frequently undergo state transformations. From an apple being cut into pieces to a butterfly emerging from its cocoon, tracking through these changes is important for understanding real-world objects and dynamics. However, existing methods often lose track of the target object after transformation, due to significant changes in object appearance. To address this limitation, we introduce the task of Track Any State: tracking objects through transformations while detecting and describing state changes, accompanied by a new benchmark dataset, VOST-TAS. To tackle this problem, we present TubeletGraph, a zero-shot system that recovers missing objects after transformation and maps out how object states are evolving over time. TubeletGraph first identifies potentially overlooked tracks, and determines whether they should be integrated based on semantic and proximity priors. Then, it reasons about the added tracks and generates a state graph describing each observed transformation. TubeletGraph achieves state-of-the-art tracking performance under transformations, while demonstrating deeper understanding of object transformations and promising capabilities in temporal grounding and semantic reasoning for complex object transformations. Code, additional results, and the benchmark dataset are available at https://tubelet-graph.github.io.