Oracle MCG: A first peek into COCO Detection Challenges

TL;DR

This paper introduces an oracle-based benchmark for the COCO detection challenge, providing a performance upper bound that highlights the difficulty of the dataset and sets a reference for future object detection research.

Contribution

It presents the first oracle-based upper bound for COCO detection, offering a benchmark to evaluate the difficulty of the dataset and the potential of current object proposal methods.

Findings

Oracle achieves AP=0.292 for segmentation

Oracle achieves AP=0.317 for bounding boxes

Highlights the challenge level of COCO dataset

Abstract

The recently presented COCO detection challenge will most probably be the reference benchmark in object detection in the next years. COCO is two orders of magnitude larger than Pascal and has four times the number of categories; so in all likelihood researchers will be faced with a number of new challenges. At this point, without any finished round of the competition, it is difficult for researchers to put their techniques in context, or in other words, to know how good their results are. In order to give a little context, this note evaluates a hypothetical object detector consisting in an oracle picking the best object proposal from a state-of-the-art technique. This oracle achieves a AP=0.292 in segmented objects and AP=0.317 in bounding boxes, showing that indeed the database is challenging, given that this value is the best one can expect if working on object proposals without refinement.