Assessing hierarchies by their consistent segmentations

TL;DR

This paper introduces new methods for specifying image segmentations from hierarchies, proposes an efficient approach to optimize segmentation quality based on the Jaccard index, and demonstrates the variability and potential of hierarchy-based segmentation quality.

Contribution

It presents novel ways to specify segmentations using hierarchy elements and an efficient method to optimize segmentation quality, providing bounds on achievable segmentation performance.

Findings

Segmentation quality varies significantly with specification methods.

Few hierarchy elements can often represent high-quality segmentations.

Efficient optimization of the Jaccard index from hierarchies is feasible.

Abstract

Current approaches to generic segmentation start by creating a hierarchy of nested image partitions and then specifying a segmentation from it. Our first contribution is to describe several ways, most of them new, for specifying segmentations using the hierarchy elements. Then, we consider the best hierarchy-induced segmentation specified by a limited number of hierarchy elements. We focus on a common quality measure for binary segmentations, the Jaccard index (also known as IoU). Optimizing the Jaccard index is highly non-trivial, and yet we propose an efficient approach for doing exactly that. This way we get algorithm-independent upper bounds on the quality of any segmentation created from the hierarchy. We found that the obtainable segmentation quality varies significantly depending on the way that the segments are specified by the hierarchy elements, and that representing a segmentation with only a few hierarchy elements is often possible. (Code is available).