Compositional Risk Minimization

TL;DR

This paper introduces compositional risk minimization (CRM), a novel training approach designed to improve model generalization to unseen attribute combinations under compositional shifts, supported by theoretical analysis and empirical results.

Contribution

The paper proposes CRM, a new method for training classifiers that better generalizes to unseen attribute combinations, with a theoretical foundation and empirical validation.

Findings

CRM improves robustness to compositional shifts

Theoretical analysis shows CRM extrapolates to affine hulls of seen data

Empirical results outperform existing methods on benchmark datasets

Abstract

Compositional generalization is a crucial step towards developing data-efficient intelligent machines that generalize in human-like ways. In this work, we tackle a challenging form of distribution shift, termed compositional shift, where some attribute combinations are completely absent at training but present in the test distribution. This shift tests the model's ability to generalize compositionally to novel attribute combinations in discriminative tasks. We model the data with flexible additive energy distributions, where each energy term represents an attribute, and derive a simple alternative to empirical risk minimization termed compositional risk minimization (CRM). We first train an additive energy classifier to predict the multiple attributes and then adjust this classifier to tackle compositional shifts. We provide an extensive theoretical analysis of CRM, where we show that our proposal extrapolates to special affine hulls of seen attribute combinations. Empirical evaluations on benchmark datasets confirms the improved robustness of CRM compared to other methods from the literature designed to tackle various forms of subpopulation shifts.