Resolving References to Objects in Photographs using the Words-As-Classifiers Model

TL;DR

This paper applies the Words-As-Classifiers model to real-world photographs for object reference resolution, demonstrating competitive performance with state-of-the-art methods while emphasizing conceptual simplicity and flexibility.

Contribution

It extends the Words-As-Classifiers model to large-scale, real-world datasets, integrating CNN features and positional data for improved reference resolution.

Findings

Achieves performance competitive with state-of-the-art methods

Utilizes CNN features combined with positional information

Demonstrates model's flexibility and conceptual simplicity

Abstract

A common use of language is to refer to visually present objects. Modelling it in computers requires modelling the link between language and perception. The "words as classifiers" model of grounded semantics views words as classifiers of perceptual contexts, and composes the meaning of a phrase through composition of the denotations of its component words. It was recently shown to perform well in a game-playing scenario with a small number of object types. We apply it to two large sets of real-world photographs that contain a much larger variety of types and for which referring expressions are available. Using a pre-trained convolutional neural network to extract image features, and augmenting these with in-picture positional information, we show that the model achieves performance competitive with the state of the art in a reference resolution task (given expression, find bounding box of its referent), while, as we argue, being conceptually simpler and more flexible.