Entailment-Preserving First-order Logic Representations in Natural Language Entailment

TL;DR

This paper introduces the EPF task and evaluation metrics for generating entailment-preserving FOL representations from natural language, proposing a specialized training method that improves entailment preservation in logical translations.

Contribution

It defines the EPF task, develops reference-free evaluation metrics, and proposes an iterative learning-to-rank method that enhances entailment preservation in FOL representations.

Findings

The proposed method improves EPR by 1.8-2.7%.

EPR@16 increases by 17.4-20.6%.

Method reduces predicate signature diversity.

Abstract

First-order logic (FOL) can represent the logical entailment semantics of natural language (NL) sentences, but determining natural language entailment using FOL remains a challenge. To address this, we propose the Entailment-Preserving FOL representations (EPF) task and introduce reference-free evaluation metrics for EPF, the Entailment-Preserving Rate (EPR) family. In EPF, one should generate FOL representations from multi-premise natural language entailment data (e.g. EntailmentBank) so that the automatic prover's result preserves the entailment labels. Experiments show that existing methods for NL-to-FOL translation struggle in EPF. To this extent, we propose a training method specialized for the task, iterative learning-to-rank, which directly optimizes the model's EPR score through a novel scoring function and a learning-to-rank objective. Our method achieves a 1.8-2.7% improvement in EPR and a 17.4-20.6% increase in EPR@16 compared to diverse baselines in three datasets. Further analyses reveal that iterative learning-to-rank effectively suppresses the arbitrariness of FOL representation by reducing the diversity of predicate signatures, and maintains strong performance across diverse inference types and out-of-domain data.