Logic and the $2$-Simplicial Transformer

TL;DR

The paper presents the 2-simplicial Transformer, an advanced neural architecture that extends attention mechanisms to higher dimensions, enhancing logical reasoning capabilities in deep reinforcement learning tasks.

Contribution

It introduces the 2-simplicial Transformer with higher-dimensional attention and demonstrates its effectiveness for logical reasoning in reinforcement learning.

Findings

Improved logical reasoning performance in RL tasks.

Higher-dimensional attention enhances entity representation updates.

The architecture generalizes traditional dot-product attention.

Abstract

We introduce the $2$-simplicial Transformer, an extension of the Transformer which includes a form of higher-dimensional attention generalising the dot-product attention, and uses this attention to update entity representations with tensor products of value vectors. We show that this architecture is a useful inductive bias for logical reasoning in the context of deep reinforcement learning.