Deep Spatial Pyramid: The Devil is Once Again in the Details

TL;DR

This paper demonstrates that careful selection of detailed factors in deep learning-based visual recognition, such as normalization and spatial pyramid design, significantly improves image classification accuracy with a simple, efficient system.

Contribution

It identifies and validates key detailed factors, including normalization and spatial pyramid design, that enhance deep learning image classification performance.

Findings

L2 matrix normalization outperforms other normalization methods.

Natural deep spatial pyramid significantly boosts accuracy.

Small Fisher Vector K values outperform larger ones.

Abstract

In this paper we show that by carefully making good choices for various detailed but important factors in a visual recognition framework using deep learning features, one can achieve a simple, efficient, yet highly accurate image classification system. We first list 5 important factors, based on both existing researches and ideas proposed in this paper. These important detailed factors include: 1) $\ell_2$ matrix normalization is more effective than unnormalized or $\ell_2$ vector normalization, 2) the proposed natural deep spatial pyramid is very effective, and 3) a very small $K$ in Fisher Vectors surprisingly achieves higher accuracy than normally used large $K$ values. Along with other choices (convolutional activations and multiple scales), the proposed DSP framework is not only intuitive and efficient, but also achieves excellent classification accuracy on many benchmark datasets. For example, DSP's accuracy on SUN397 is 59.78%, significantly higher than previous state-of-the-art (53.86%).