Learning Semantic Structure through First-Order-Logic Translation

TL;DR

This paper investigates how transformer-based language models can learn predicate argument structures, comparing question answering and first-order logic translation tasks, with a focus on their generalization abilities and the effectiveness of finetuning versus prompting.

Contribution

It demonstrates that finetuning large language models on FOL translation tasks enhances their ability to understand predicate argument structures compared to other methods.

Findings

FOL translation improves predicate-argument extraction in LLMs.

Finetuning yields better generalization than prompting.

LLMs excel at FOL translation over Q/A for semantic structure learning.

Abstract

In this paper, we study whether transformer-based language models can extract predicate argument structure from simple sentences. We firstly show that language models sometimes confuse which predicates apply to which objects. To mitigate this, we explore two tasks: question answering (Q/A), and first order logic (FOL) translation, and two regimes, prompting and finetuning. In FOL translation, we finetune several large language models on synthetic datasets designed to gauge their generalization abilities. For Q/A, we finetune encoder models like BERT and RoBERTa and use prompting for LLMs. The results show that FOL translation for LLMs is better suited to learn predicate argument structure.