A Testbed for Cross-Dataset Analysis

TL;DR

This paper introduces a comprehensive testbed that consolidates twelve image datasets into a single corpus, facilitating large-scale analysis of dataset biases and their impact on visual recognition system generalization.

Contribution

It creates a unified dataset repository and analysis framework to study dataset biases across multiple visual recognition datasets, aiding future research.

Findings

Organized twelve datasets into a single corpus

Provided a feature repository for dataset analysis

Facilitated large-scale bias analysis in visual recognition

Abstract

Since its beginning visual recognition research has tried to capture the huge variability of the visual world in several image collections. The number of available datasets is still progressively growing together with the amount of samples per object category. However, this trend does not correspond directly to an increasing in the generalization capabilities of the developed recognition systems. Each collection tends to have its specific characteristics and to cover just some aspects of the visual world: these biases often narrow the effect of the methods defined and tested separately over each image set. Our work makes a first step towards the analysis of the dataset bias problem on a large scale. We organize twelve existing databases in a unique corpus and we present the visual community with a useful feature repository for future research.