Exploring Graph Representations of Logical Forms for Language Modeling

TL;DR

This paper introduces a graph-based logical form language model (LFLM) that is more data-efficient and outperforms traditional text-based models like BERT on downstream tasks, demonstrating the potential of logical form representations.

Contribution

The paper presents GFoLDS, a novel pretrained language model over graph representations of logical forms, showing improved data efficiency and performance over textual models.

Findings

GFoLDS outperforms BERT on downstream tasks.

LFLMs leverage inherent linguistic knowledge from graph structures.

Model performance scales with more data and parameters.

Abstract

We make the case for language models over logical forms (LFLMs), arguing that such models are more data-efficient than their textual counterparts. To that end, we introduce the Graph-based Formal-Logical Distributional Semantics (GFoLDS) prototype, a pretrained LM over graph representations of logical forms, as a proof-of-concept of LFLMs. Using GFoLDS, we present strong experimental evidence that LFLMs can leverage the built-in, basic linguistic knowledge inherent in such models to immediately begin learning more complex patterns. On downstream tasks, we show that GFoLDS vastly outperforms textual, transformer LMs (BERT) pretrained on the same data, indicating that LFLMs can learn with substantially less data than models over plain text. Furthermore, we show that the performance of this model is likely to scale with additional parameters and pretraining data, suggesting the viability of LFLMs in real-world applications.