TED: A Tolerant Edit Distance for Segmentation Evaluation

TL;DR

The paper introduces Tolerant Edit Distance (TED), a new segmentation error measure that accounts for tolerable boundary errors, provides intuitive error quantification, and estimates the manual correction time needed.

Contribution

The novel TED measure incorporates application-dependent tolerances, uses integer linear programming for minimal error calculation, and offers error localization and classification capabilities.

Findings

TED effectively counts topological errors in 3D neuron segmentation.

TED ignores small boundary shifts, focusing on significant errors.

TED provides a time-to-fix estimate for segmentation corrections.

Abstract

In this paper, we present a novel error measure to compare a segmentation against ground truth. This measure, which we call Tolerant Edit Distance (TED), is motivated by two observations: (1) Some errors, like small boundary shifts, are tolerable in practice. Which errors are tolerable is application dependent and should be a parameter of the measure. (2) Non-tolerable errors have to be corrected manually. The time needed to do so should be reflected by the error measure. Using integer linear programming, the TED finds the minimal weighted sum of split and merge errors exceeding a given tolerance criterion, and thus provides a time-to-fix estimate. In contrast to commonly used measures like Rand index or variation of information, the TED (1) does not count small, but tolerable, differences, (2) provides intuitive numbers, (3) gives a time-to-fix estimate, and (4) can localize and classify the type of errors. By supporting both isotropic and anisotropic volumes and having a flexible tolerance criterion, the TED can be adapted to different requirements. On example segmentations for 3D neuron segmentation, we demonstrate that the TED is capable of counting topological errors, while ignoring small boundary shifts.