Linguistic Frameworks Go Toe-to-Toe at Neuro-Symbolic Language Modeling

TL;DR

This paper investigates how linguistic graph representations, especially semantic constituency structures, can enhance neural language models, revealing their varying effectiveness across parts of speech and formalism types.

Contribution

It demonstrates that semantic constituency graphs outperform other formalism types in improving language modeling within a neuro-symbolic framework.

Findings

Semantic constituency structures are most beneficial for language modeling.

Effects of graph formalism vary significantly by part-of-speech.

Results suggest promising directions for neuro-symbolic language modeling.

Abstract

We examine the extent to which, in principle, linguistic graph representations can complement and improve neural language modeling. With an ensemble setup consisting of a pretrained Transformer and ground-truth graphs from one of 7 different formalisms, we find that, overall, semantic constituency structures are most useful to language modeling performance -- outpacing syntactic constituency structures as well as syntactic and semantic dependency structures. Further, effects vary greatly depending on part-of-speech class. In sum, our findings point to promising tendencies in neuro-symbolic language modeling and invite future research quantifying the design choices made by different formalisms.