Composition-Incremental Learning for Compositional Generalization

TL;DR

This paper introduces a new incremental learning framework for compositional generalization in computer vision, enabling models to progressively learn and improve on new compositions in a continual manner, addressing real-world data challenges.

Contribution

It proposes the Composition-Incremental Learning (CompIL) framework, including a benchmark pipeline and a pseudo-replay method with visual synthesis and primitive distillation.

Findings

Effective in improving compositional generalization over time

Benchmark datasets demonstrate the framework's superiority

Maintains primitive representation alignment during learning

Abstract

Compositional generalization has achieved substantial progress in computer vision on pre-collected training data. Nonetheless, real-world data continually emerges, with possible compositions being nearly infinite, long-tailed, and not entirely visible. Thus, an ideal model is supposed to gradually improve the capability of compositional generalization in an incremental manner. In this paper, we explore Composition-Incremental Learning for Compositional Generalization (CompIL) in the context of the compositional zero-shot learning (CZSL) task, where models need to continually learn new compositions, intending to improve their compositional generalization capability progressively. To quantitatively evaluate CompIL, we develop a benchmark construction pipeline leveraging existing datasets, yielding MIT-States-CompIL and C-GQA-CompIL. Furthermore, we propose a pseudo-replay framework utilizing a visual synthesizer to synthesize visual representations of learned compositions and a linguistic primitive distillation mechanism to maintain aligned primitive representations across the learning process. Extensive experiments demonstrate the effectiveness of the proposed framework.