Seeing through bag-of-visual-word glasses: towards understanding quantization effects in feature extraction methods

TL;DR

This paper investigates how quantization in local feature extraction affects visual recognition, using novel inversion techniques to visualize and compare the information loss to human perception.

Contribution

It introduces a new visualization method to analyze the impact of feature quantization on information retention in bag-of-visual-words models.

Findings

Quantization causes significant information loss in local features.

Visualizations reveal the extent of detail preserved at different codebook sizes.

Human recognition performance correlates with the amount of visual information retained.

Abstract

Vector-quantized local features frequently used in bag-of-visual-words approaches are the backbone of popular visual recognition systems due to both their simplicity and their performance. Despite their success, bag-of-words-histograms basically contain low-level image statistics (e.g., number of edges of different orientations). The question remains how much visual information is "lost in quantization" when mapping visual features to code words? To answer this question, we present an in-depth analysis of the effect of local feature quantization on human recognition performance. Our analysis is based on recovering the visual information by inverting quantized local features and presenting these visualizations with different codebook sizes to human observers. Although feature inversion techniques are around for quite a while, to the best of our knowledge, our technique is the first visualizing especially the effect of feature quantization. Thereby, we are now able to compare single steps in common image classification pipelines to human counterparts.