Voxel-informed Language Grounding

TL;DR

The paper introduces Voxel-informed Language Grounder (VLG), a model that uses 3D voxel maps from visual input to improve language grounding accuracy in 3D environments, achieving state-of-the-art results.

Contribution

VLG is the first model to incorporate volumetric 3D voxel information for language grounding, significantly enhancing accuracy on SNARE.

Findings

VLG achieves top performance on SNARE leaderboard.

VLG improves grounding accuracy by 2.0% absolute.

Voxel information enhances language grounding in 3D tasks.

Abstract

Natural language applied to natural 2D images describes a fundamentally 3D world. We present the Voxel-informed Language Grounder (VLG), a language grounding model that leverages 3D geometric information in the form of voxel maps derived from the visual input using a volumetric reconstruction model. We show that VLG significantly improves grounding accuracy on SNARE, an object reference game task. At the time of writing, VLG holds the top place on the SNARE leaderboard, achieving SOTA results with a 2.0% absolute improvement.