What is Learned in Visually Grounded Neural Syntax Acquisition

TL;DR

This paper analyzes a visually grounded neural syntax model, revealing that simpler versions and lexical cues like noun concreteness are key to its performance, challenging assumptions about complex syntactic learning.

Contribution

It demonstrates that a simplified model and lexical signals can achieve similar or better performance, questioning the necessity of complex syntactic reasoning in visual grounding.

Findings

Simpler models perform as well or better than complex ones.

Noun concreteness is a primary factor in model predictions.

Complex syntactic reasoning is less critical than lexical cues.

Abstract

Visual features are a promising signal for learning bootstrap textual models. However, blackbox learning models make it difficult to isolate the specific contribution of visual components. In this analysis, we consider the case study of the Visually Grounded Neural Syntax Learner (Shi et al., 2019), a recent approach for learning syntax from a visual training signal. By constructing simplified versions of the model, we isolate the core factors that yield the model's strong performance. Contrary to what the model might be capable of learning, we find significantly less expressive versions produce similar predictions and perform just as well, or even better. We also find that a simple lexical signal of noun concreteness plays the main role in the model's predictions as opposed to more complex syntactic reasoning.