Learning Generalizable Feature Fields for Mobile Manipulation

TL;DR

This paper introduces GeFF, a neural scene representation that unifies navigation and manipulation tasks for mobile robots, leveraging generative view synthesis and CLIP-based semantic alignment for real-time, open-vocabulary scene understanding.

Contribution

The work presents GeFF, a novel scene-level neural feature field that combines geometry and semantics for mobile manipulation, enabling real-time, open-vocabulary tasks with improved efficiency.

Findings

Outperforms point-based baselines in runtime and storage-accuracy trade-offs.

Enables semantics-aware navigation and articulated object manipulation.

Demonstrates effectiveness on a quadrupedal robot with manipulation capabilities.

Abstract

An open problem in mobile manipulation is how to represent objects and scenes in a unified manner so that robots can use both for navigation and manipulation. The latter requires capturing intricate geometry while understanding fine-grained semantics, whereas the former involves capturing the complexity inherent at an expansive physical scale. In this work, we present GeFF (Generalizable Feature Fields), a scene-level generalizable neural feature field that acts as a unified representation for both navigation and manipulation that performs in real-time. To do so, we treat generative novel view synthesis as a pre-training task, and then align the resulting rich scene priors with natural language via CLIP feature distillation. We demonstrate the effectiveness of this approach by deploying GeFF on a quadrupedal robot equipped with a manipulator. We quantitatively evaluate GeFF's ability for open-vocabulary object-/part-level manipulation and show that GeFF outperforms point-based baselines in runtime and storage-accuracy trade-offs, with qualitative examples of semantics-aware navigation and articulated object manipulation.