OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completion

TL;DR

OAKINK2 is a comprehensive dataset capturing bimanual object manipulation in complex tasks, structured into three abstraction levels, supporting advanced task understanding, reconstruction, and motion synthesis.

Contribution

The paper introduces OAKINK2, a novel dataset with multi-view images and annotations, and a task-oriented framework utilizing LLMs and motion models for complex task completion.

Findings

OAKINK2 enables detailed analysis of bimanual manipulation.

The framework successfully decomposes complex tasks into primitive actions.

Motion models generate realistic bimanual hand movements.

Abstract

We present OAKINK2, a dataset of bimanual object manipulation tasks for complex daily activities. In pursuit of constructing the complex tasks into a structured representation, OAKINK2 introduces three level of abstraction to organize the manipulation tasks: Affordance, Primitive Task, and Complex Task. OAKINK2 features on an object-centric perspective for decoding the complex tasks, treating them as a sequence of object affordance fulfillment. The first level, Affordance, outlines the functionalities that objects in the scene can afford, the second level, Primitive Task, describes the minimal interaction units that humans interact with the object to achieve its affordance, and the third level, Complex Task, illustrates how Primitive Tasks are composed and interdependent. OAKINK2 dataset provides multi-view image streams and precise pose annotations for the human body, hands and various interacting objects. This extensive collection supports applications such as interaction reconstruction and motion synthesis. Based on the 3-level abstraction of OAKINK2, we explore a task-oriented framework for Complex Task Completion (CTC). CTC aims to generate a sequence of bimanual manipulation to achieve task objectives. Within the CTC framework, we employ Large Language Models (LLMs) to decompose the complex task objectives into sequences of Primitive Tasks and have developed a Motion Fulfillment Model that generates bimanual hand motion for each Primitive Task. OAKINK2 datasets and models are available at https://oakink.net/v2.