A Minimalist Dataset for Systematic Generalization of Perception, Syntax, and Semantics

TL;DR

This paper introduces HINT, a minimalistic dataset designed to evaluate machine learning models' ability to generalize systematically across perception, syntax, and semantics in arithmetic tasks, revealing current models' limitations and potential improvements.

Contribution

The paper presents HINT, a new dataset for testing systematic generalization in perception, syntax, and semantics, along with extensive experiments highlighting current models' challenges and the effectiveness of chain of thought prompting.

Findings

Models struggle with long-range syntactic dependencies.

Scaling dataset and model size yields limited improvements.

Chain of thought prompting significantly improves GPT-3 zero-shot performance.

Abstract

Inspired by humans' exceptional ability to master arithmetic and generalize to new problems, we present a new dataset, Handwritten arithmetic with INTegers (HINT), to examine machines' capability of learning generalizable concepts at three levels: perception, syntax, and semantics. In HINT, machines are tasked with learning how concepts are perceived from raw signals such as images (i.e., perception), how multiple concepts are structurally combined to form a valid expression (i.e., syntax), and how concepts are realized to afford various reasoning tasks (i.e., semantics), all in a weakly supervised manner. Focusing on systematic generalization, we carefully design a five-fold test set to evaluate both the interpolation and the extrapolation of learned concepts w.r.t. the three levels. Further, we design a few-shot learning split to determine whether or not models can rapidly learn new concepts and generalize them to more complex scenarios. To comprehend existing models' limitations, we undertake extensive experiments with various sequence-to-sequence models, including RNNs, Transformers, and GPT-3 (with the chain of thought prompting). The results indicate that current models struggle to extrapolate to long-range syntactic dependency and semantics. Models exhibit a considerable gap toward human-level generalization when evaluated with new concepts in a few-shot setting. Moreover, we discover that it is infeasible to solve HINT by merely scaling up the dataset and the model size; this strategy contributes little to the extrapolation of syntax and semantics. Finally, in zero-shot GPT-3 experiments, the chain of thought prompting exhibits impressive results and significantly boosts the test accuracy. We believe the HINT dataset and the experimental findings are of great interest to the learning community on systematic generalization.