Reasoning About Pragmatics with Neural Listeners and Speakers

TL;DR

This paper introduces a neural model that combines learned semantics and inference-driven pragmatics to generate contextually appropriate descriptions, achieving higher success in referring expression tasks than previous methods.

Contribution

It presents a novel neural listener and speaker architecture that reason about pragmatics without explicit pragmatic annotations during training.

Findings

Achieves 81% success in referring expression game

Outperforms existing techniques with 69% success rate

Uses only ordinary captions for training

Abstract

We present a model for pragmatically describing scenes, in which contrastive behavior results from a combination of inference-driven pragmatics and learned semantics. Like previous learned approaches to language generation, our model uses a simple feature-driven architecture (here a pair of neural "listener" and "speaker" models) to ground language in the world. Like inference-driven approaches to pragmatics, our model actively reasons about listener behavior when selecting utterances. For training, our approach requires only ordinary captions, annotated _without_ demonstration of the pragmatic behavior the model ultimately exhibits. In human evaluations on a referring expression game, our approach succeeds 81% of the time, compared to a 69% success rate using existing techniques.