Object Manipulation via Visual Target Localization

TL;DR

This paper introduces m-VOLE, a method enabling embodied AI agents to locate and track objects in 3D for manipulation tasks, significantly improving success rates and robustness in realistic, noisy environments.

Contribution

The paper presents m-VOLE, a novel approach that estimates object locations in 3D and maintains tracking even when objects are out of view, advancing real-world applicability of embodied AI.

Findings

3x improvement in success rate over baseline models

Robustness to noise in depth perception and localization

Relaxation of idealized perception assumptions

Abstract

Object manipulation is a critical skill required for Embodied AI agents interacting with the world around them. Training agents to manipulate objects, poses many challenges. These include occlusion of the target object by the agent's arm, noisy object detection and localization, and the target frequently going out of view as the agent moves around in the scene. We propose Manipulation via Visual Object Location Estimation (m-VOLE), an approach that explores the environment in search for target objects, computes their 3D coordinates once they are located, and then continues to estimate their 3D locations even when the objects are not visible, thus robustly aiding the task of manipulating these objects throughout the episode. Our evaluations show a massive 3x improvement in success rate over a model that has access to the same sensory suite but is trained without the object location estimator, and our analysis shows that our agent is robust to noise in depth perception and agent localization. Importantly, our proposed approach relaxes several assumptions about idealized localization and perception that are commonly employed by recent works in embodied AI -- an important step towards training agents for object manipulation in the real world.