Hierarchical Visual Primitive Experts for Compositional Zero-Shot Learning

TL;DR

This paper introduces Composition Transformer (CoT), a hierarchical framework for compositional zero-shot learning that explicitly models attribute-object contextuality and addresses data imbalance, achieving state-of-the-art results.

Contribution

The paper proposes a novel hierarchical framework with object and attribute experts, and a minority attribute augmentation method for improved compositional zero-shot learning.

Findings

Achieves state-of-the-art performance on MIT-States, C-GQA, and VAW-CZSL benchmarks.

Effectively models contextuality between attributes and objects.

Addresses data imbalance with virtual sample augmentation.

Abstract

Compositional zero-shot learning (CZSL) aims to recognize unseen compositions with prior knowledge of known primitives (attribute and object). Previous works for CZSL often suffer from grasping the contextuality between attribute and object, as well as the discriminability of visual features, and the long-tailed distribution of real-world compositional data. We propose a simple and scalable framework called Composition Transformer (CoT) to address these issues. CoT employs object and attribute experts in distinctive manners to generate representative embeddings, using the visual network hierarchically. The object expert extracts representative object embeddings from the final layer in a bottom-up manner, while the attribute expert makes attribute embeddings in a top-down manner with a proposed object-guided attention module that models contextuality explicitly. To remedy biased prediction caused by imbalanced data distribution, we develop a simple minority attribute augmentation (MAA) that synthesizes virtual samples by mixing two images and oversampling minority attribute classes. Our method achieves SoTA performance on several benchmarks, including MIT-States, C-GQA, and VAW-CZSL. We also demonstrate the effectiveness of CoT in improving visual discrimination and addressing the model bias from the imbalanced data distribution. The code is available at https://github.com/HanjaeKim98/CoT.