Systematic Generalization with Edge Transformers

TL;DR

Edge Transformer introduces a novel model combining vector edge states and triangular attention, significantly improving systematic generalization in natural language understanding tasks over traditional Transformers.

Contribution

The paper proposes Edge Transformer, integrating edge-based vector states and a triangular attention mechanism inspired by logic programming for enhanced generalization.

Findings

Outperforms baseline Transformers in relational reasoning

Achieves better results in semantic parsing tasks

Improves dependency parsing accuracy

Abstract

Recent research suggests that systematic generalization in natural language understanding remains a challenge for state-of-the-art neural models such as Transformers and Graph Neural Networks. To tackle this challenge, we propose Edge Transformer, a new model that combines inspiration from Transformers and rule-based symbolic AI. The first key idea in Edge Transformers is to associate vector states with every edge, that is, with every pair of input nodes -- as opposed to just every node, as it is done in the Transformer model. The second major innovation is a triangular attention mechanism that updates edge representations in a way that is inspired by unification from logic programming. We evaluate Edge Transformer on compositional generalization benchmarks in relational reasoning, semantic parsing, and dependency parsing. In all three settings, the Edge Transformer outperforms Relation-aware, Universal and classical Transformer baselines.