PokeFlex: A Real-World Dataset of Volumetric Deformable Objects for Robotics

TL;DR

PokeFlex is a comprehensive real-world dataset of volumetric deformable objects with multimodal data, enabling advanced manipulation and reconstruction tasks in robotics, and supporting the deployment of traditional control methods in real-world scenarios.

Contribution

The paper introduces PokeFlex, a novel multimodal dataset of deformable objects captured with a professional volumetric system, facilitating research in 3D reconstruction and manipulation.

Findings

Achieved state-of-the-art multi-object mesh reconstruction from multimodal data.

Demonstrated the dataset's utility in online 3D mesh reconstruction tasks.

Provided a diverse set of deformable objects with detailed annotations.

Abstract

Data-driven methods have shown great potential in solving challenging manipulation tasks; however, their application in the domain of deformable objects has been constrained, in part, by the lack of data. To address this lack, we propose PokeFlex, a dataset featuring real-world multimodal data that is paired and annotated. The modalities include 3D textured meshes, point clouds, RGB images, and depth maps. Such data can be leveraged for several downstream tasks, such as online 3D mesh reconstruction, and it can potentially enable underexplored applications such as the real-world deployment of traditional control methods based on mesh simulations. To deal with the challenges posed by real-world 3D mesh reconstruction, we leverage a professional volumetric capture system that allows complete 360{\deg} reconstruction. PokeFlex consists of 18 deformable objects with varying stiffness and shapes. Deformations are generated by dropping objects onto a flat surface or by poking the objects with a robot arm. Interaction wrenches and contact locations are also reported for the latter case. Using different data modalities, we demonstrated a use case for our dataset training models that, given the novelty of the multimodal nature of Pokeflex, constitute the state-of-the-art in multi-object online template-based mesh reconstruction from multimodal data, to the best of our knowledge. We refer the reader to our website ( https://pokeflex-dataset.github.io/ ) for further demos and examples.