The Wilhelm Tell Dataset of Affordance Demonstrations

TL;DR

This paper introduces a new video dataset capturing human demonstrations of household tasks to improve robot perception of affordances, including first- and third-person views and task metadata.

Contribution

The work provides a novel, multi-view affordance demonstration dataset with metadata, enabling better training of perception systems for household robot tasks.

Findings

Dataset includes 7 hours of diverse human demonstrations.

Includes first- and third-person perspectives and task metadata.

Facilitates studying preparatory maneuvers for tasks.

Abstract

Affordances - i.e. possibilities for action that an environment or objects in it provide - are important for robots operating in human environments to perceive. Existing approaches train such capabilities on annotated static images or shapes. This work presents a novel dataset for affordance learning of common household tasks. Unlike previous approaches, our dataset consists of video sequences demonstrating the tasks from first- and third-person perspectives, along with metadata about the affordances that are manifested in the task, and is aimed towards training perception systems to recognize affordance manifestations. The demonstrations were collected from several participants and in total record about seven hours of human activity. The variety of task performances also allows studying preparatory maneuvers that people may perform for a task, such as how they arrange their task space, which is also relevant for collaborative service robots.