Scalable Evaluation and Neural Models for Compositional Generalization

TL;DR

This paper introduces a scalable evaluation framework and neural models that significantly improve compositional generalization in vision tasks, addressing current limitations in benchmarks and model inductive biases.

Contribution

It presents a unified evaluation protocol, extensive empirical analysis of vision models, and Attribute Invariant Networks that enhance compositional generalization with fewer parameters.

Findings

Achieved a 23.43% accuracy improvement over baselines.

Reduced parameter overhead from 600% to 16%.

Evaluated over 5000 models on compositional generalization tasks.

Abstract

Compositional generalization-a key open challenge in modern machine learning-requires models to predict unknown combinations of known concepts. However, assessing compositional generalization remains a fundamental challenge due to the lack of standardized evaluation protocols and the limitations of current benchmarks, which often favor efficiency over rigor. At the same time, general-purpose vision architectures lack the necessary inductive biases, and existing approaches to endow them compromise scalability. As a remedy, this paper introduces: 1) a rigorous evaluation framework that unifies and extends previous approaches while reducing computational requirements from combinatorial to constant; 2) an extensive and modern evaluation on the status of compositional generalization in supervised vision backbones, training more than 5000 models; 3) Attribute Invariant Networks, a class of models establishing a new Pareto frontier in compositional generalization, achieving a 23.43% accuracy improvement over baselines while reducing parameter overhead from 600% to 16% compared to fully disentangled counterparts. Our code is available at https://github.com/IBM/scalable-compositional-generalization.