Structural generalization is hard for sequence-to-sequence models

TL;DR

Seq2seq models struggle with generalizing to unseen linguistic structures across various NLP tasks, but neurosymbolic models with built-in linguistic knowledge can often overcome this limitation.

Contribution

This paper provides evidence that the generalization challenge of seq2seq models is widespread and highlights the potential of neurosymbolic models to address it.

Findings

Seq2seq models perform poorly on unseen linguistic structures.

Neurosymbolic models with linguistic knowledge improve generalization.

Limitations of seq2seq models are consistent across multiple NLP tasks.

Abstract

Sequence-to-sequence (seq2seq) models have been successful across many NLP tasks, including ones that require predicting linguistic structure. However, recent work on compositional generalization has shown that seq2seq models achieve very low accuracy in generalizing to linguistic structures that were not seen in training. We present new evidence that this is a general limitation of seq2seq models that is present not just in semantic parsing, but also in syntactic parsing and in text-to-text tasks, and that this limitation can often be overcome by neurosymbolic models that have linguistic knowledge built in. We further report on some experiments that give initial answers on the reasons for these limitations.