Recognizing Manipulation Actions from State-Transformations

TL;DR

This paper introduces a method for recognizing manipulation actions by analyzing object state transitions, which are more visually apparent than actions in still frames and complement traditional spatio-temporal recognition.

Contribution

It proposes a novel approach using object state transitions and a state transition matrix for action recognition, validated on the EPIC kitchen challenge.

Findings

Effective recognition of manipulation actions using object state transitions.

Complementary to existing spatio-temporal methods.

Validated results on EPIC kitchen dataset.

Abstract

Manipulation actions transform objects from an initial state into a final state. In this paper, we report on the use of object state transitions as a mean for recognizing manipulation actions. Our method is inspired by the intuition that object states are visually more apparent than actions from a still frame and thus provide information that is complementary to spatio-temporal action recognition. We start by defining a state transition matrix that maps action labels into a pre-state and a post-state. From each keyframe, we learn appearance models of objects and their states. Manipulation actions can then be recognized from the state transition matrix. We report results on the EPIC kitchen action recognition challenge.