Learning Interpretable Spatial Operations in a Rich 3D Blocks World

TL;DR

This paper introduces a new dataset and neural model for mapping natural language instructions to complex spatial actions in a 3D blocks world, enabling more interpretable and accurate spatial understanding.

Contribution

It presents a novel dataset with rich natural language descriptions of 3D spatial operations and a neural architecture that learns interpretable spatial operations from this data.

Findings

Achieved competitive results on 3D spatial instruction tasks

Discovered an interpretable set of spatial operations automatically

Enhanced understanding of complex spatial language in 3D environments

Abstract

In this paper, we study the problem of mapping natural language instructions to complex spatial actions in a 3D blocks world. We first introduce a new dataset that pairs complex 3D spatial operations to rich natural language descriptions that require complex spatial and pragmatic interpretations such as "mirroring", "twisting", and "balancing". This dataset, built on the simulation environment of Bisk, Yuret, and Marcu (2016), attains language that is significantly richer and more complex, while also doubling the size of the original dataset in the 2D environment with 100 new world configurations and 250,000 tokens. In addition, we propose a new neural architecture that achieves competitive results while automatically discovering an inventory of interpretable spatial operations (Figure 5)