Learning Subclass Representations for Visually-varied Image Classification

TL;DR

This paper introduces a subclass-representation method for image classification that leverages tag co-occurrence and content to improve accuracy on diverse classes, demonstrated on a large-scale Flickr dataset.

Contribution

The paper proposes a novel subclass-based representation approach that combines image content and tag co-occurrence, enhancing classification of visually diverse classes.

Findings

Improved classification performance on a large-scale Flickr dataset.

Subclass representations outperform direct top-class modeling.

Method effectively exploits tag co-occurrence information.

Abstract

In this paper, we present a subclass-representation approach that predicts the probability of a social image belonging to one particular class. We explore the co-occurrence of user-contributed tags to find subclasses with a strong connection to the top level class. We then project each image on to the resulting subclass space to generate a subclass representation for the image. The novelty of the approach is that subclass representations make use of not only the content of the photos themselves, but also information on the co-occurrence of their tags, which determines membership in both subclasses and top-level classes. The novelty is also that the images are classified into smaller classes, which have a chance of being more visually stable and easier to model. These subclasses are used as a latent space and images are represented in this space by their probability of relatedness to all of the subclasses. In contrast to approaches directly modeling each top-level class based on the image content, the proposed method can exploit more information for visually diverse classes. The approach is evaluated on a set of $2$ million photos with 10 classes, released by the Multimedia 2013 Yahoo! Large-scale Flickr-tag Image Classification Grand Challenge. Experiments show that the proposed system delivers sound performance for visually diverse classes compared with methods that directly model top classes.