Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architectures

TL;DR

This paper evaluates various architectures and pre-training techniques for improving compositional generalization in semantic parsing, finding that MLM pre-training with intermediate representations achieves state-of-the-art results on CFQ.

Contribution

It demonstrates that MLM pre-training with intermediate representations significantly enhances compositional generalization, surpassing specialized architectures on CFQ.

Findings

MLM pre-training rivals SCAN-inspired architectures on primitive holdout splits.

Pre-training significantly improves performance on complex compositional tasks.

Architectures designed for compositional generalization show limited improvements.

Abstract

While mainstream machine learning methods are known to have limited ability to compositionally generalize, new architectures and techniques continue to be proposed to address this limitation. We investigate state-of-the-art techniques and architectures in order to assess their effectiveness in improving compositional generalization in semantic parsing tasks based on the SCAN and CFQ datasets. We show that masked language model (MLM) pre-training rivals SCAN-inspired architectures on primitive holdout splits. On a more complex compositional task, we show that pre-training leads to significant improvements in performance vs. comparable non-pre-trained models, whereas architectures proposed to encourage compositional generalization on SCAN or in the area of algorithm learning fail to lead to significant improvements. We establish a new state of the art on the CFQ compositional generalization benchmark using MLM pre-training together with an intermediate representation.