Independent Prototype Propagation for Zero-Shot Compositionality

TL;DR

ProtoProp is a novel graph-based method that learns independent prototypes for objects and attributes, propagates them to form compositional prototypes, and improves zero-shot recognition without external data.

Contribution

It introduces ProtoProp, a prototype propagation graph that learns independent object and attribute prototypes for better zero-shot compositional reasoning.

Findings

Outperforms state-of-the-art in generalized compositional zero-shot recognition

Effective on both synthetic and real-world datasets

Highlights importance of independent prototype learning and propagation

Abstract

Humans are good at compositional zero-shot reasoning; someone who has never seen a zebra before could nevertheless recognize one when we tell them it looks like a horse with black and white stripes. Machine learning systems, on the other hand, usually leverage spurious correlations in the training data, and while such correlations can help recognize objects in context, they hurt generalization. To be able to deal with underspecified datasets while still leveraging contextual clues during classification, we propose ProtoProp, a novel prototype propagation graph method. First we learn prototypical representations of objects (e.g., zebra) that are conditionally independent w.r.t. their attribute labels (e.g., stripes) and vice versa. Next we propagate the independent prototypes through a compositional graph, to learn compositional prototypes of novel attribute-object combinations that reflect the dependencies of the target distribution. The method does not rely on any external data, such as class hierarchy graphs or pretrained word embeddings. We evaluate our approach on AO-Clever, a synthetic and strongly visual dataset with clean labels, and UT-Zappos, a noisy real-world dataset of fine-grained shoe types. We show that in the generalized compositional zero-shot setting we outperform state-of-the-art results, and through ablations we show the importance of each part of the method and their contribution to the final results.