Dependency Induction Through the Lens of Visual Perception

TL;DR

This paper introduces an unsupervised model that uses visual and lexical cues to improve grammar induction, significantly enhancing dependency and constituency parsing performance over text-only models.

Contribution

It proposes a novel joint learning approach for constituency and dependency grammars leveraging word concreteness and visual cues, advancing visually grounded syntax models.

Findings

Concreteness improves dependency grammar learning, increasing DAS by over 50%.

Visual semantic role labels enhance constituency parsing accuracy.

The model outperforms existing visually grounded models with smaller grammars.

Abstract

Most previous work on grammar induction focuses on learning phrasal or dependency structure purely from text. However, because the signal provided by text alone is limited, recently introduced visually grounded syntax models make use of multimodal information leading to improved performance in constituency grammar induction. However, as compared to dependency grammars, constituency grammars do not provide a straightforward way to incorporate visual information without enforcing language-specific heuristics. In this paper, we propose an unsupervised grammar induction model that leverages word concreteness and a structural vision-based heuristic to jointly learn constituency-structure and dependency-structure grammars. Our experiments find that concreteness is a strong indicator for learning dependency grammars, improving the direct attachment score (DAS) by over 50\% as compared to state-of-the-art models trained on pure text. Next, we propose an extension of our model that leverages both word concreteness and visual semantic role labels in constituency and dependency parsing. Our experiments show that the proposed extension outperforms the current state-of-the-art visually grounded models in constituency parsing even with a smaller grammar size.