Evaluation for Uncertain Image Classification and Segmentation

TL;DR

This paper introduces a new evaluation method for image classification and segmentation that considers both the results and the certainty levels provided by human experts, addressing limitations of traditional evaluation approaches.

Contribution

It proposes a novel evaluation framework that incorporates expert certainty levels into the assessment of classification and segmentation algorithms.

Findings

Effective evaluation of seabed characterization algorithm

Incorporates expert certainty into performance metrics

Addresses uncertainty in human-labeled data

Abstract

Each year, numerous segmentation and classification algorithms are invented or reused to solve problems where machine vision is needed. Generally, the efficiency of these algorithms is compared against the results given by one or many human experts. However, in many situations, the location of the real boundaries of the objects as well as their classes are not known with certainty by the human experts. Furthermore, only one aspect of the segmentation and classification problem is generally evaluated. In this paper we present a new evaluation method for classification and segmentation of image, where we take into account both the classification and segmentation results as well as the level of certainty given by the experts. As a concrete example of our method, we evaluate an automatic seabed characterization algorithm based on sonar images.