Structural Transfer Learning in NL-to-Bash Semantic Parsers

TL;DR

This paper investigates how structural similarities between different NLP tasks affect transfer learning, revealing that lexical overlap is key and that more pre-training compute does not always enhance transfer to semantic parsing.

Contribution

It introduces a methodology to quantify structural overlap between NLP tasks and applies it to analyze transfer learning in NL-to-Bash semantic parsing.

Findings

Structural overlap between NL-to-Bash and SQL is strong.

Lexical alignment largely explains transfer success.

More pre-training compute does not guarantee better transfer.

Abstract

Large-scale pre-training has made progress in many fields of natural language processing, though little is understood about the design of pre-training datasets. We propose a methodology for obtaining a quantitative understanding of structural overlap between machine translation tasks. We apply our methodology to the natural language to Bash semantic parsing task (NLBash) and show that it is largely reducible to lexical alignment. We also find that there is strong structural overlap between NLBash and natural language to SQL. Additionally, we perform a study varying compute expended during pre-training on the English to German machine translation task and find that more compute expended during pre-training does not always correspond semantic representations with stronger transfer to NLBash.