'Part'ly first among equals: Semantic part-based benchmarking for state-of-the-art object recognition systems

TL;DR

This paper introduces a new benchmarking dataset and procedure focusing on object parts and challenging conditions to better differentiate top object recognition systems' robustness beyond traditional error metrics.

Contribution

The paper presents PPSS-12, a new dataset with varied object-part visibility, and a semantic part-based benchmarking method that assesses classifiers' robustness to occlusion and context variations.

Findings

Top classifiers show similar error rates on standard benchmarks.

The new method reveals differences in robustness to occlusion and context.

Benchmarking with PPSS-12 provides finer differentiation among classifiers.

Abstract

An examination of object recognition challenge leaderboards (ILSVRC, PASCAL-VOC) reveals that the top-performing classifiers typically exhibit small differences amongst themselves in terms of error rate/mAP. To better differentiate the top performers, additional criteria are required. Moreover, the (test) images, on which the performance scores are based, predominantly contain fully visible objects. Therefore, `harder' test images, mimicking the challenging conditions (e.g. occlusion) in which humans routinely recognize objects, need to be utilized for benchmarking. To address the concerns mentioned above, we make two contributions. First, we systematically vary the level of local object-part content, global detail and spatial context in images from PASCAL VOC 2010 to create a new benchmarking dataset dubbed PPSS-12. Second, we propose an object-part based benchmarking procedure which quantifies classifiers' robustness to a range of visibility and contextual settings. The benchmarking procedure relies on a semantic similarity measure that naturally addresses potential semantic granularity differences between the category labels in training and test datasets, thus eliminating manual mapping. We use our procedure on the PPSS-12 dataset to benchmark top-performing classifiers trained on the ILSVRC-2012 dataset. Our results show that the proposed benchmarking procedure enables additional differentiation among state-of-the-art object classifiers in terms of their ability to handle missing content and insufficient object detail. Given this capability for additional differentiation, our approach can potentially supplement existing benchmarking procedures used in object recognition challenge leaderboards.