Compositional Generalization with Grounded Language Models

TL;DR

This paper evaluates how well grounded language models generalize compositionally when combined with knowledge graphs, revealing current limitations and proposing new evaluation methods.

Contribution

It introduces a controlled evaluation framework and datasets to assess compositional generalization in grounded language models, highlighting their struggles with unseen combinations and lengths.

Findings

Models struggle with unseen sequence lengths

Difficulty in generalizing to novel component combinations

Current methods have limited compositional generalization capabilities

Abstract

Grounded language models use external sources of information, such as knowledge graphs, to meet some of the general challenges associated with pre-training. By extending previous work on compositional generalization in semantic parsing, we allow for a controlled evaluation of the degree to which these models learn and generalize from patterns in knowledge graphs. We develop a procedure for generating natural language questions paired with knowledge graphs that targets different aspects of compositionality and further avoids grounding the language models in information already encoded implicitly in their weights. We evaluate existing methods for combining language models with knowledge graphs and find them to struggle with generalization to sequences of unseen lengths and to novel combinations of seen base components. While our experimental results provide some insight into the expressive power of these models, we hope our work and released datasets motivate future research on how to better combine language models with structured knowledge representations.