Precise Affordance Annotation for Egocentric Action Video Datasets

TL;DR

This paper introduces a precise annotation scheme for object affordance in egocentric videos, addressing existing dataset issues and improving affordance recognition for human-object interaction tasks.

Contribution

It proposes a novel annotation scheme combining motor actions and grasp types, and applies it to enhance affordance labeling in the EPIC-KITCHENS dataset.

Findings

Models trained with new annotations can distinguish affordance from mechanical actions.

The scheme reduces confusion between affordance, object functionality, and goal-related actions.

Annotations improve affordance recognition accuracy in egocentric videos.

Abstract

Object affordance is an important concept in human-object interaction, providing information on action possibilities based on human motor capacity and objects' physical property thus benefiting tasks such as action anticipation and robot imitation learning. However, existing datasets often: 1) mix up affordance with object functionality; 2) confuse affordance with goal-related action; and 3) ignore human motor capacity. This paper proposes an efficient annotation scheme to address these issues by combining goal-irrelevant motor actions and grasp types as affordance labels and introducing the concept of mechanical action to represent the action possibilities between two objects. We provide new annotations by applying this scheme to the EPIC-KITCHENS dataset and test our annotation with tasks such as affordance recognition. We qualitatively verify that models trained with our annotation can distinguish affordance and mechanical actions.