The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers

TL;DR

This paper shows that simple adjustments to transformer configurations, such as embedding scaling and positional embeddings, significantly enhance their systematic generalization across various datasets, highlighting the importance of proper validation sets.

Contribution

The study demonstrates that basic model configuration tweaks can drastically improve transformer systematic generalization, emphasizing the need for proper validation strategies.

Findings

Transformers' performance improves from 50% to 85% on PCFG.

Relative positional embedding achieves 100% accuracy on SCAN length split.

Model improvements are often invisible on IID data splits.

Abstract

Recently, many datasets have been proposed to test the systematic generalization ability of neural networks. The companion baseline Transformers, typically trained with default hyper-parameters from standard tasks, are shown to fail dramatically. Here we demonstrate that by revisiting model configurations as basic as scaling of embeddings, early stopping, relative positional embedding, and Universal Transformer variants, we can drastically improve the performance of Transformers on systematic generalization. We report improvements on five popular datasets: SCAN, CFQ, PCFG, COGS, and Mathematics dataset. Our models improve accuracy from 50% to 85% on the PCFG productivity split, and from 35% to 81% on COGS. On SCAN, relative positional embedding largely mitigates the EOS decision problem (Newman et al., 2020), yielding 100% accuracy on the length split with a cutoff at 26. Importantly, performance differences between these models are typically invisible on the IID data split. This calls for proper generalization validation sets for developing neural networks that generalize systematically. We publicly release the code to reproduce our results.