How hard can it be? Estimating the difficulty of visual search in an image

TL;DR

This paper introduces a deep learning-based method to estimate the difficulty of visual search tasks in images, using human annotations and image features, with applications in object localization and classification.

Contribution

The study presents a novel approach combining human annotations and deep features to predict image difficulty, improving ranking accuracy and generalization to unseen classes.

Findings

Deep features outperform simple image properties in difficulty prediction

The model correctly ranks 75% of image pairs by difficulty

Difficulty scores enhance weakly supervised object localization and classification

Abstract

We address the problem of estimating image difficulty defined as the human response time for solving a visual search task. We collect human annotations of image difficulty for the PASCAL VOC 2012 data set through a crowd-sourcing platform. We then analyze what human interpretable image properties can have an impact on visual search difficulty, and how accurate are those properties for predicting difficulty. Next, we build a regression model based on deep features learned with state of the art convolutional neural networks and show better results for predicting the ground-truth visual search difficulty scores produced by human annotators. Our model is able to correctly rank about 75% image pairs according to their difficulty score. We also show that our difficulty predictor generalizes well to new classes not seen during training. Finally, we demonstrate that our predicted difficulty scores are useful for weakly supervised object localization (8% improvement) and semi-supervised object classification (1% improvement).