H-SAUR: Hypothesize, Simulate, Act, Update, and Repeat for Understanding Object Articulations from Interactions

TL;DR

H-SAUR is a probabilistic framework enabling autonomous agents to understand and manipulate articulated objects through hypothesis generation, simulation, and iterative updating, significantly improving efficiency and accuracy without requiring training data.

Contribution

The paper introduces H-SAUR, a novel probabilistic framework that models hypotheses about object articulations and guides exploration, outperforming existing methods without training data.

Findings

H-SAUR outperforms state-of-the-art methods on PartNet-Mobility.

H-SAUR effectively solves multi-step puzzles in the PuzzleBoxes benchmark.

Incorporating learned priors improves test-time efficiency.

Abstract

The world is filled with articulated objects that are difficult to determine how to use from vision alone, e.g., a door might open inwards or outwards. Humans handle these objects with strategic trial-and-error: first pushing a door then pulling if that doesn't work. We enable these capabilities in autonomous agents by proposing "Hypothesize, Simulate, Act, Update, and Repeat" (H-SAUR), a probabilistic generative framework that simultaneously generates a distribution of hypotheses about how objects articulate given input observations, captures certainty over hypotheses over time, and infer plausible actions for exploration and goal-conditioned manipulation. We compare our model with existing work in manipulating objects after a handful of exploration actions, on the PartNet-Mobility dataset. We further propose a novel PuzzleBoxes benchmark that contains locked boxes that require multiple steps to solve. We show that the proposed model significantly outperforms the current state-of-the-art articulated object manipulation framework, despite using zero training data. We further improve the test-time efficiency of H-SAUR by integrating a learned prior from learning-based vision models.