Systematic Generalization on gSCAN: What is Nearly Solved and What is Next?

TL;DR

This paper evaluates the gSCAN benchmark for grounded language understanding, showing that Transformer models perform well on many tasks, but still face fundamental challenges in systematic generalization and data efficiency, leading to new task proposals.

Contribution

It demonstrates the strong performance of Transformer models on gSCAN and introduces new challenging tasks to address remaining systematic generalization issues.

Findings

Transformer models outperform specialized methods on many gSCAN splits

Remaining errors highlight fundamental systematic generalization challenges

Current models are data inefficient for the narrow command scope

Abstract

We analyze the grounded SCAN (gSCAN) benchmark, which was recently proposed to study systematic generalization for grounded language understanding. First, we study which aspects of the original benchmark can be solved by commonly used methods in multi-modal research. We find that a general-purpose Transformer-based model with cross-modal attention achieves strong performance on a majority of the gSCAN splits, surprisingly outperforming more specialized approaches from prior work. Furthermore, our analysis suggests that many of the remaining errors reveal the same fundamental challenge in systematic generalization of linguistic constructs regardless of visual context. Second, inspired by this finding, we propose challenging new tasks for gSCAN by generating data to incorporate relations between objects in the visual environment. Finally, we find that current models are surprisingly data inefficient given the narrow scope of commands in gSCAN, suggesting another challenge for future work.