PokeNet: Learning Kinematic Models of Articulated Objects from Human Observations

TL;DR

PokeNet is an end-to-end framework that learns articulation models of unknown objects from a single human demonstration, accurately predicting joint parameters, manipulation order, and joint states without prior object knowledge.

Contribution

It introduces a novel method that estimates articulation models from minimal data, handling occlusions and manipulation sequences for diverse and unseen objects.

Findings

Improves joint axis and state estimation accuracy by over 27%.

Works effectively in both simulation and real-world environments.

Handles occlusions and manipulation order without prior object knowledge.

Abstract

Articulation modeling enables robots to learn joint parameters of articulated objects for effective manipulation which can then be used downstream for skill learning or planning. Existing approaches often rely on prior knowledge about the objects, such as the number or type of joints. Some of these approaches also fail to recover occluded joints that are only revealed during interaction. Others require large numbers of multi-view images for every object, which is impractical in real-world settings. Furthermore, prior works neglect the order of manipulations, which is essential for many multi-DoF objects where one joint must be operated before another, such as a dishwasher. We introduce PokeNet, an end-to-end framework that estimates articulation models from a single human demonstration without prior object knowledge. Given a sequence of point cloud observations of a human manipulating an unknown object, PokeNet predicts joint parameters, infers manipulation order, and tracks joint states over time. PokeNet outperforms existing state-of-the-art methods, improving joint axis and state estimation accuracy by an average of over 27% across diverse objects, including novel and unseen categories. We demonstrate these gains in both simulation and real-world environments.