OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interaction

TL;DR

OakInk is a comprehensive large-scale knowledge repository that captures human-object interactions and affordances, enabling improved understanding and generation of hand-object interactions for robotics and AI applications.

Contribution

The paper introduces OakInk, the first large-scale, multi-modal knowledge base combining object affordances and human interactions, with a novel transfer method Tink for virtual interaction synthesis.

Findings

OakInk contains 50,000 annotated hand-object interactions.

Benchmark results demonstrate improved pose estimation and grasp generation.

Practical applications include intent-based interaction and handover generation.

Abstract

Learning how humans manipulate objects requires machines to acquire knowledge from two perspectives: one for understanding object affordances and the other for learning human's interactions based on the affordances. Even though these two knowledge bases are crucial, we find that current databases lack a comprehensive awareness of them. In this work, we propose a multi-modal and rich-annotated knowledge repository, OakInk, for visual and cognitive understanding of hand-object interactions. We start to collect 1,800 common household objects and annotate their affordances to construct the first knowledge base: Oak. Given the affordance, we record rich human interactions with 100 selected objects in Oak. Finally, we transfer the interactions on the 100 recorded objects to their virtual counterparts through a novel method: Tink. The recorded and transferred hand-object interactions constitute the second knowledge base: Ink. As a result, OakInk contains 50,000 distinct affordance-aware and intent-oriented hand-object interactions. We benchmark OakInk on pose estimation and grasp generation tasks. Moreover, we propose two practical applications of OakInk: intent-based interaction generation and handover generation. Our datasets and source code are publicly available at https://github.com/lixiny/OakInk.