Weakly Supervised Learning of Objects, Attributes and their Associations

TL;DR

This paper introduces a weakly supervised non-parametric Bayesian model that learns object-attribute associations from image-level labels, enabling detailed image descriptions without requiring detailed annotations.

Contribution

It presents a novel weakly supervised approach to model object-attribute associations, reducing the need for detailed annotations and improving scalability.

Findings

Performs comparably to strongly supervised models on image description tasks

Effectively models object-attribute associations from weak labels

Enables image segmentation and localization based on learned associations

Abstract

When humans describe images they tend to use combinations of nouns and adjectives, corresponding to objects and their associated attributes respectively. To generate such a description automatically, one needs to model objects, attributes and their associations. Conventional methods require strong annotation of object and attribute locations, making them less scalable. In this paper, we model object-attribute associations from weakly labelled images, such as those widely available on media sharing sites (e.g. Flickr), where only image-level labels (either object or attributes) are given, without their locations and associations. This is achieved by introducing a novel weakly supervised non-parametric Bayesian model. Once learned, given a new image, our model can describe the image, including objects, attributes and their associations, as well as their locations and segmentation. Extensive experiments on benchmark datasets demonstrate that our weakly supervised model performs at par with strongly supervised models on tasks such as image description and retrieval based on object-attribute associations.