Making Transformers Solve Compositional Tasks

TL;DR

This paper investigates how different Transformer design choices affect their ability to generalize compositionally in NLP tasks, leading to improved configurations that outperform previous models on key benchmarks.

Contribution

The study systematically explores Transformer design space, identifying configurations that enhance compositional generalization and achieve state-of-the-art results on multiple benchmarks.

Findings

Identified Transformer configurations with significantly improved compositional generalization.

Achieved state-of-the-art results on COGS and PCFG benchmarks.

Demonstrated the impact of inductive biases on model generalization.

Abstract

Several studies have reported the inability of Transformer models to generalize compositionally, a key type of generalization in many NLP tasks such as semantic parsing. In this paper we explore the design space of Transformer models showing that the inductive biases given to the model by several design decisions significantly impact compositional generalization. Through this exploration, we identified Transformer configurations that generalize compositionally significantly better than previously reported in the literature in a diverse set of compositional tasks, and that achieve state-of-the-art results in a semantic parsing compositional generalization benchmark (COGS), and a string edit operation composition benchmark (PCFG).