g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Tasks

TL;DR

g3D-LF is a pre-trained 3D representation model that encodes multi-scale, multi-view features aligned with language, enabling improved embodied task performance in unseen environments.

Contribution

We propose g3D-LF, a novel 3D-language feature field model trained on large-scale data for generalizable embodied task applications.

Findings

Effective in Vision-and-Language Navigation tasks

Enables zero-shot object navigation

Improves situated question answering accuracy

Abstract

We introduce Generalizable 3D-Language Feature Fields (g3D-LF), a 3D representation model pre-trained on large-scale 3D-language dataset for embodied tasks. Our g3D-LF processes posed RGB-D images from agents to encode feature fields for: 1) Novel view representation predictions from any position in the 3D scene; 2) Generations of BEV maps centered on the agent; 3) Querying targets using multi-granularity language within the above-mentioned representations. Our representation can be generalized to unseen environments, enabling real-time construction and dynamic updates. By volume rendering latent features along sampled rays and integrating semantic and spatial relationships through multiscale encoders, our g3D-LF produces representations at different scales and perspectives, aligned with multi-granularity language, via multi-level contrastive learning. Furthermore, we prepare a large-scale 3D-language dataset to align the representations of the feature fields with language. Extensive experiments on Vision-and-Language Navigation under both Panorama and Monocular settings, Zero-shot Object Navigation, and Situated Question Answering tasks highlight the significant advantages and effectiveness of our g3D-LF for embodied tasks.