Neural Abstructions: Abstractions that Support Construction for Grounded Language Learning

TL;DR

This paper introduces neural abstructions, a novel approach combining the strengths of semantic parsers and end-to-end models to enable flexible, expressive language grounding for virtual agents through user-defined constraints.

Contribution

It proposes neural abstructions, a new method for constraining generative models, allowing users to build adaptable semantic parsers from minimal examples.

Findings

Users built a semantic parser for Minecraft modifications.

The parser's redefinition rate increased to 28% over 191 exchanges.

The approach balances flexibility and expressiveness in language grounding.

Abstract

Although virtual agents are increasingly situated in environments where natural language is the most effective mode of interaction with humans, these exchanges are rarely used as an opportunity for learning. Leveraging language interactions effectively requires addressing limitations in the two most common approaches to language grounding: semantic parsers built on top of fixed object categories are precise but inflexible and end-to-end models are maximally expressive, but fickle and opaque. Our goal is to develop a system that balances the strengths of each approach so that users can teach agents new instructions that generalize broadly from a single example. We introduce the idea of neural abstructions: a set of constraints on the inference procedure of a label-conditioned generative model that can affect the meaning of the label in context. Starting from a core programming language that operates over abstructions, users can define increasingly complex mappings from natural language to actions. We show that with this method a user population is able to build a semantic parser for an open-ended house modification task in Minecraft. The semantic parser that results is both flexible and expressive: the percentage of utterances sourced from redefinitions increases steadily over the course of 191 total exchanges, achieving a final value of 28%.