Semantic Diversity versus Visual Diversity in Visual Dictionaries

TL;DR

This paper investigates the relative importance of semantic versus visual diversity in creating effective visual dictionaries for image classification, concluding visual diversity is more crucial.

Contribution

It provides an empirical evaluation showing that visual diversity outweighs semantic diversity in dictionary quality for BoVW models.

Findings

Visual diversity is more important than semantic diversity.

Good dictionaries can be built with visually diverse images regardless of semantic coverage.

Reducing the need for semantic sampling simplifies dictionary creation.

Abstract

Visual dictionaries are a critical component for image classification/retrieval systems based on the bag-of-visual-words (BoVW) model. Dictionaries are usually learned without supervision from a training set of images sampled from the collection of interest. However, for large, general-purpose, dynamic image collections (e.g., the Web), obtaining a representative sample in terms of semantic concepts is not straightforward. In this paper, we evaluate the impact of semantics in the dictionary quality, aiming at verifying the importance of semantic diversity in relation visual diversity for visual dictionaries. In the experiments, we vary the amount of classes used for creating the dictionary and then compute different BoVW descriptors, using multiple codebook sizes and different coding and pooling methods (standard BoVW and Fisher Vectors). Results for image classification show that as visual dictionaries are based on low-level visual appearances, visual diversity is more important than semantic diversity. Our conclusions open the opportunity to alleviate the burden in generating visual dictionaries as we need only a visually diverse set of images instead of the whole collection to create a good dictionary.