Compositional Generalisation with Structured Reordering and Fertility Layers

TL;DR

This paper introduces a neural model with structured reordering and fertility layers that significantly improves compositional generalisation in semantic parsing tasks, outperforming traditional seq2seq models on complex, longer examples.

Contribution

The paper proposes a novel differentiable fertility layer combined with reordering, inspired by grammar-based models, enhancing compositional generalisation in neural networks.

Findings

Outperforms seq2seq models on compositional splits

Effective on longer, more complex examples

Favorsably compares to other compositional models

Abstract

Seq2seq models have been shown to struggle with compositional generalisation, i.e. generalising to new and potentially more complex structures than seen during training. Taking inspiration from grammar-based models that excel at compositional generalisation, we present a flexible end-to-end differentiable neural model that composes two structural operations: a fertility step, which we introduce in this work, and a reordering step based on previous work (Wang et al., 2021). To ensure differentiability, we use the expected value of each step. Our model outperforms seq2seq models by a wide margin on challenging compositional splits of realistic semantic parsing tasks that require generalisation to longer examples. It also compares favourably to other models targeting compositional generalisation.