An Empirical Study of Uncertainty in Polygon Annotation and the Impact of Quality Assurance

TL;DR

This paper investigates the inherent uncertainty in polygon annotations for instance segmentation and evaluates how quality assurance procedures influence annotation reliability, highlighting factors like scene and shape complexity.

Contribution

It provides an empirical analysis of polygon annotation uncertainty and demonstrates the importance of review procedures in improving annotation quality.

Findings

Annotation reliability depends on review procedures.

Scene and shape complexity affect annotation quality.

Quality assurance reduces annotation uncertainty.

Abstract

Polygons are a common annotation format used for quickly annotating objects in instance segmentation tasks. However, many real-world annotation projects request near pixel-perfect labels. While strict pixel guidelines may appear to be the solution to a successful project, practitioners often fail to assess the feasibility of the work requested, and overlook common factors that may challenge the notion of quality. This paper aims to examine and quantify the inherent uncertainty for polygon annotations and the role that quality assurance plays in minimizing its effect. To this end, we conduct an analysis on multi-rater polygon annotations for several objects from the MS-COCO dataset. The results demonstrate that the reliability of a polygon annotation is dependent on a reviewing procedure, as well as the scene and shape complexity.