A small note on variation in segmentation annotations

TL;DR

This paper demonstrates that manual segmentation annotations vary significantly among annotators and that removing inconsistent annotators can reduce variation, revealing meaningful biological structures.

Contribution

It highlights the variability in manual segmentation annotations and proposes a method to improve reference standards by removing inconsistent annotators.

Findings

Removing worst annotators reduces segmentation variation.

Remaining annotations capture semantically meaningful structures.

Manual annotations are not absolute ground truth but reference standards.

Abstract

We report on the results of a small crowdsourcing experiment conducted at a workshop on machine learning for segmentation held at the Danish Bio Imaging network meeting 2020. During the workshop we asked participants to manually segment mitochondria in three 2D patches. The aim of the experiment was to illustrate that manual annotations should not be seen as the ground truth, but as a reference standard that is subject to substantial variation. In this note we show how the large variation we observed in the segmentations can be reduced by removing the annotators with worst pair-wise agreement. Having removed the annotators with worst performance, we illustrate that the remaining variance is semantically meaningful and can be exploited to obtain segmentations of cell boundary and cell interior.