Categorising the World into Local Climate Zones -- Towards Quantifying Labelling Uncertainty for Machine Learning Models

TL;DR

This paper models and analyzes the uncertainty in labeling satellite images for climate zone classification, using a multinomial mixture model to understand sources of ambiguity in expert evaluations.

Contribution

It introduces a multinomial mixture model to quantify and analyze labeling uncertainty in satellite image classification, accounting for expert heterogeneity and image origin.

Findings

Insights into sources of label uncertainty

Analysis of expert heterogeneity impacts

Understanding of class ambiguity in remote sensing

Abstract

Image classification is often prone to labelling uncertainty. To generate suitable training data, images are labelled according to evaluations of human experts. This can result in ambiguities, which will affect subsequent models. In this work, we aim to model the labelling uncertainty in the context of remote sensing and the classification of satellite images. We construct a multinomial mixture model given the evaluations of multiple experts. This is based on the assumption that there is no ambiguity of the image class, but apparently in the experts' opinion about it. The model parameters can be estimated by a stochastic Expectation Maximization algorithm. Analysing the estimates gives insights into sources of label uncertainty. Here, we focus on the general class ambiguity, the heterogeneity of experts, and the origin city of the images. The results are relevant for all machine learning applications where image classification is pursued and labelling is subject to humans.