Enhancing Transformers for Generalizable First-Order Logical Entailment

TL;DR

This paper investigates how transformers perform in generalizable first-order logical reasoning, identifies architectural limitations, and introduces TEGA, a logic-aware model that enhances reasoning capabilities and outperforms previous methods.

Contribution

It provides a comprehensive analysis of transformers' reasoning abilities, reveals design mismatches, and proposes TEGA to improve logical entailment performance.

Findings

Transformers outperform previous methods in knowledge graph query answering.

Input query syntax and architecture significantly impact reasoning performance.

Positional encoding mismatches affect transformer reasoning capabilities.

Abstract

Transformers, as the fundamental deep learning architecture, have demonstrated great capability in reasoning. This paper studies the generalizable first-order logical reasoning ability of transformers with their parameterized knowledge and how to improve it. Transformers' capability of first-order reasoning is further captured by whether they can conduct first-order logical entailment, which is quantitatively measured by their performance in answering knowledge graph queries. We establish the connections between (1) two types of distribution shifts studied in out-of-distribution generalization and (2) unseen knowledge and query settings discussed in the task of knowledge graph query answering, which makes it possible to characterize the fine-grained generalizability. Results on our comprehensive dataset showed that transformers \textit{outperform} previous methods designed particularly for this task and provided detailed empirical evidence about the impact of the input query syntax, token embedding, and transformer architectures on their reasoning capability. Interestingly, our results revealed the mismatch of positional encoding and other design choices of transformer architectures in previous practices. Motivated by this, we propose TEGA, a logic-aware architecture that significantly improves the performance in generalizable first-order logical entailment.