Context Determines Optimal Architecture in Materials Segmentation

TL;DR

This paper introduces a cross-modal evaluation framework for materials image segmentation, revealing that the optimal neural network architecture varies with imaging modality and providing tools for deployment reliability and interpretability.

Contribution

It presents a systematic evaluation of segmentation architectures across multiple imaging modalities and offers deployment guidance and interpretability tools for materials characterization.

Findings

UNet performs best for high-contrast 2D images

DeepLabv3+ is preferred for challenging cases

Framework includes out-of-distribution detection and feature explanations

Abstract

Segmentation architectures are typically benchmarked on single imaging modalities, obscuring deployment-relevant performance variations: an architecture optimal for one modality may underperform on another. We present a cross-modal evaluation framework for materials image segmentation spanning SEM, AFM, XCT, and optical microscopy. Our evaluation of six encoder-decoder combinations across seven datasets reveals that optimal architectures vary systematically by context: UNet excels for high-contrast 2D imaging while DeepLabv3+ is preferred for the hardest cases. The framework also provides deployment feedback via out-of-distribution detection and counterfactual explanations that reveal which microstructural features drive predictions. Together, the architecture guidance, reliability signals, and interpretability tools address a practical gap in materials characterization, where researchers lack tools to select architectures for their specific imaging setup or assess when models can be trusted on new samples.