Attributes as Operators: Factorizing Unseen Attribute-Object Compositions

TL;DR

This paper introduces a novel method modeling visual attributes as operators, enabling better recognition of unseen attribute-object combinations by explicitly separating attributes from objects in a semantic embedding.

Contribution

The paper proposes modeling attributes as operators within a semantic space, improving generalization to unseen attribute-object compositions and incorporating attribute effects with new regularizers.

Findings

Significant improvements over state-of-the-art on two datasets

Robust recognition of unseen attribute-object compositions

Generalization to unseen objects during training

Abstract

We present a new approach to modeling visual attributes. Prior work casts attributes in a similar role as objects, learning a latent representation where properties (e.g., sliced) are recognized by classifiers much in the way objects (e.g., apple) are. However, this common approach fails to separate the attributes observed during training from the objects with which they are composed, making it ineffectual when encountering new attribute-object compositions. Instead, we propose to model attributes as operators. Our approach learns a semantic embedding that explicitly factors out attributes from their accompanying objects, and also benefits from novel regularizers expressing attribute operators' effects (e.g., blunt should undo the effects of sharp). Not only does our approach align conceptually with the linguistic role of attributes as modifiers, but it also generalizes to recognize unseen compositions of objects and attributes. We validate our approach on two challenging datasets and demonstrate significant improvements over the state-of-the-art. In addition, we show that not only can our model recognize unseen compositions robustly in an open-world setting, it can also generalize to compositions where objects themselves were unseen during training.