Naturally Supervised 3D Visual Grounding with Language-Regularized Concept Learners

TL;DR

This paper introduces LARC, a language-regularized neuro-symbolic model for 3D visual grounding that learns effectively from only scene and QA pairs, avoiding dense supervision.

Contribution

It proposes a novel language regularization approach for neuro-symbolic models, improving 3D visual grounding without dense supervision and enabling better reasoning capabilities.

Findings

LARC outperforms prior methods in naturally supervised 3D visual grounding.

Demonstrates zero-shot composition and transferability.

Enhances data efficiency in 3D visual reasoning.

Abstract

3D visual grounding is a challenging task that often requires direct and dense supervision, notably the semantic label for each object in the scene. In this paper, we instead study the naturally supervised setting that learns from only 3D scene and QA pairs, where prior works underperform. We propose the Language-Regularized Concept Learner (LARC), which uses constraints from language as regularization to significantly improve the accuracy of neuro-symbolic concept learners in the naturally supervised setting. Our approach is based on two core insights: the first is that language constraints (e.g., a word's relation to another) can serve as effective regularization for structured representations in neuro-symbolic models; the second is that we can query large language models to distill such constraints from language properties. We show that LARC improves performance of prior works in naturally supervised 3D visual grounding, and demonstrates a wide range of 3D visual reasoning capabilities-from zero-shot composition, to data efficiency and transferability. Our method represents a promising step towards regularizing structured visual reasoning frameworks with language-based priors, for learning in settings without dense supervision.