CURI: A Benchmark for Productive Concept Learning Under Uncertainty

TL;DR

CURI is a new benchmark designed to evaluate how well models can learn and reason about complex, uncertain, and compositional concepts in a few-shot setting, addressing limitations of traditional classification tasks.

Contribution

The paper introduces CURI, a comprehensive benchmark for systematic generalization and reasoning under uncertainty across multiple modalities, with a model-independent measure of compositionality difficulty.

Findings

Models show significant room for improvement on CURI tasks.

Extensive evaluations reveal challenges in out-of-distribution generalization.

Benchmark spans images, schemas, and sounds, encouraging diverse modeling approaches.

Abstract

Humans can learn and reason under substantial uncertainty in a space of infinitely many concepts, including structured relational concepts ("a scene with objects that have the same color") and ad-hoc categories defined through goals ("objects that could fall on one's head"). In contrast, standard classification benchmarks: 1) consider only a fixed set of category labels, 2) do not evaluate compositional concept learning and 3) do not explicitly capture a notion of reasoning under uncertainty. We introduce a new few-shot, meta-learning benchmark, Compositional Reasoning Under Uncertainty (CURI) to bridge this gap. CURI evaluates different aspects of productive and systematic generalization, including abstract understandings of disentangling, productive generalization, learning boolean operations, variable binding, etc. Importantly, it also defines a model-independent "compositionality gap" to evaluate the difficulty of generalizing out-of-distribution along each of these axes. Extensive evaluations across a range of modeling choices spanning different modalities (image, schemas, and sounds), splits, privileged auxiliary concept information, and choices of negatives reveal substantial scope for modeling advances on the proposed task. All code and datasets will be available online.