Improving Object Permanence using Agent Actions and Reasoning

TL;DR

This paper enhances object permanence in robots by integrating agent action reasoning, significantly improving performance in complex scenarios and demonstrating practical applicability in real-world industrial tasks.

Contribution

It introduces a novel approach that leverages agent actions to infer hidden object states, improving both rule-based and neural models for object permanence.

Findings

Action-annotated models outperform non-annotated ones in synthetic datasets.

The approach improves object localization accuracy in complex scenarios.

Robots successfully apply the method in real industrial environments.

Abstract

Object permanence in psychology means knowing that objects still exist even if they are no longer visible. It is a crucial concept for robots to operate autonomously in uncontrolled environments. Existing approaches learn object permanence from low-level perception, but perform poorly on more complex scenarios, like when objects are contained and carried by others. Knowledge about manipulation actions performed on an object prior to its disappearance allows us to reason about its location, e.g., that the object has been placed in a carrier. In this paper we argue that object permanence can be improved when the robot uses knowledge about executed actions and describe an approach to infer hidden object states from agent actions. We show that considering agent actions not only improves rule-based reasoning models but also purely neural approaches, showing its general applicability. Then, we conduct quantitative experiments on a snitch localization task using a dataset of 1,371 synthesized videos, where we compare the performance of different object permanence models with and without action annotations. We demonstrate that models with action annotations can significantly increase performance of both neural and rule-based approaches. Finally, we evaluate the usability of our approach in real-world applications by conducting qualitative experiments with two Universal Robots (UR5 and UR16e) in both lab and industrial settings. The robots complete benchmark tasks for a gearbox assembly and demonstrate the object permanence capabilities with real sensor data in an industrial environment.