Implementing Derivations of Definite Logic Programs with Self-Attention Networks

TL;DR

This paper demonstrates that self-attention networks within transformer models can implement logical inference processes, revealing the potential for large language models to perform logical reasoning beyond natural language semantics.

Contribution

It shows that hierarchical self-attention and feed-forward networks can realize top-down and bottom-up logical derivations, highlighting LLMs' inherent logical inference capabilities.

Findings

Self-attention networks can implement top-down logical derivations.

Self-attention networks can also realize bottom-up derivations.

Results suggest LLMs inherently possess logical inference abilities.

Abstract

In this paper we propose that a restricted version of logical inference can be implemented with self-attention networks. We are aiming at showing that LLMs (Large Language Models) constructed with transformer networks can make logical inferences. We would reveal the potential of LLMs by analyzing self-attention networks, which are main components of transformer networks. Our approach is not based on semantics of natural languages but operations of logical inference. %point of view. We show that hierarchical constructions of self-attention networks with feed forward networks (FFNs) can implement top-down derivations for a class of logical formulae. We also show bottom-up derivations are also implemented for the same class. We believe that our results show that LLMs implicitly have the power of logical inference.