Differentiable Parsing and Visual Grounding of Natural Language Instructions for Object Placement

TL;DR

This paper introduces ParaGon, a fully differentiable framework that combines parsing and graph neural networks to improve grounding of natural language instructions for object placement, handling compositionality and ambiguity effectively.

Contribution

ParaGon uniquely integrates a parsing algorithm with a probabilistic graph neural network, enabling end-to-end learning for robust language grounding in object placement tasks.

Findings

Outperforms existing methods on object placement benchmarks.

Effectively handles ambiguous and complex language instructions.

Demonstrates robustness through end-to-end training.

Abstract

We present a new method, PARsing And visual GrOuNding (ParaGon), for grounding natural language in object placement tasks. Natural language generally describes objects and spatial relations with compositionality and ambiguity, two major obstacles to effective language grounding. For compositionality, ParaGon parses a language instruction into an object-centric graph representation to ground objects individually. For ambiguity, ParaGon uses a novel particle-based graph neural network to reason about object placements with uncertainty. Essentially, ParaGon integrates a parsing algorithm into a probabilistic, data-driven learning framework. It is fully differentiable and trained end-to-end from data for robustness against complex, ambiguous language input.