Deep Robust Subjective Visual Property Prediction in Crowdsourcing

TL;DR

This paper introduces a deep probabilistic model for predicting subjective visual properties from crowdsourced pairwise comparisons, effectively handling noisy annotations and outliers to improve prediction accuracy.

Contribution

It develops a novel deep probabilistic framework combining SVP prediction and outlier detection, enabling robust learning from sparse, noisy crowdsourced data.

Findings

Outperforms existing methods on benchmark datasets

Effectively detects annotation outliers

Works well with sparse annotations

Abstract

The problem of estimating subjective visual properties (SVP) of images (e.g., Shoes A is more comfortable than B) is gaining rising attention. Due to its highly subjective nature, different annotators often exhibit different interpretations of scales when adopting absolute value tests. Therefore, recent investigations turn to collect pairwise comparisons via crowdsourcing platforms. However, crowdsourcing data usually contains outliers. For this purpose, it is desired to develop a robust model for learning SVP from crowdsourced noisy annotations. In this paper, we construct a deep SVP prediction model which not only leads to better detection of annotation outliers but also enables learning with extremely sparse annotations. Specifically, we construct a comparison multi-graph based on the collected annotations, where different labeling results correspond to edges with different directions between two vertexes. Then, we propose a generalized deep probabilistic framework which consists of an SVP prediction module and an outlier modeling module that work collaboratively and are optimized jointly. Extensive experiments on various benchmark datasets demonstrate that our new approach guarantees promising results.