Emergent Predication Structure in Hidden State Vectors of Neural Readers

TL;DR

This paper demonstrates that neural reading comprehension models develop internal representations resembling predication structures, with hidden states encoding semantic properties and entities, revealing an emergent linguistic structure.

Contribution

The study provides evidence that neural readers spontaneously develop predication-like structures in their hidden states, advancing understanding of their internal semantic representations.

Findings

Hidden state vectors encode atomic formulas with predicates and constants.

Neural readers exhibit emergent predication structures.

Supports the idea of structured semantic representations in neural models.

Abstract

A significant number of neural architectures for reading comprehension have recently been developed and evaluated on large cloze-style datasets. We present experiments supporting the emergence of "predication structure" in the hidden state vectors of these readers. More specifically, we provide evidence that the hidden state vectors represent atomic formulas $\Phi[c]$ where $\Phi$ is a semantic property (predicate) and $c$ is a constant symbol entity identifier.