Efficiently Crowdsourcing Visual Importance with Punch-Hole Annotation

TL;DR

This paper presents a new punch-hole crowdsourcing method for efficiently identifying important regions in images, reducing noise and improving reliability compared to traditional gaze or mouse-based annotations.

Contribution

The paper introduces a punch-hole annotation technique that standardizes and streamlines crowdsourced importance labeling, enhancing efficiency and accuracy.

Findings

Punch-hole labeling effectively identifies critical image regions.

The method reduces annotation noise in crowdsourcing.

Preliminary results show promising application in visualization research.

Abstract

We introduce a novel crowdsourcing method for identifying important areas in graphical images through punch-hole labeling. Traditional methods, such as gaze trackers and mouse-based annotations, which generate continuous data, can be impractical in crowdsourcing scenarios. They require many participants, and the outcome data can be noisy. In contrast, our method first segments the graphical image with a grid and drops a portion of the patches (punch holes). Then, we iteratively ask the labeler to validate each annotation with holes, narrowing down the annotation only having the most important area. This approach aims to reduce annotation noise in crowdsourcing by standardizing the annotations while enhancing labeling efficiency and reliability. Preliminary findings from fundamental charts demonstrate that punch-hole labeling can effectively pinpoint critical regions. This also highlights its potential for broader application in visualization research, particularly in studying large-scale users' graphical perception. Our future work aims to enhance the algorithm to achieve faster labeling speed and prove its utility through large-scale experiments.