Maybe you don't need a U-Net: convolutional feature upsampling for materials micrograph segmentation

TL;DR

This paper introduces a convolutional upsampling method for foundation model features that enhances micrograph segmentation efficiency and quality without additional training, outperforming traditional methods in speed and label efficiency.

Contribution

It presents a novel convolutional network for upsampling patch-based features, enabling high-quality segmentation of large, complex microscopy images without further training.

Findings

Effective segmentation of micrographs with fewer labels

Faster segmentation process compared to traditional methods

Ability to distinguish fine features like cracks

Abstract

Feature foundation models - usually vision transformers - offer rich semantic descriptors of images, useful for downstream tasks such as (interactive) segmentation and object detection. For computational efficiency these descriptors are often patch-based, and so struggle to represent the fine features often present in micrographs; they also struggle with the large image sizes present in materials and biological image analysis. In this work, we train a convolutional neural network to upsample low-resolution (i.e, large patch size) foundation model features with reference to the input image. We apply this upsampler network (without any further training) to efficiently featurise and then segment a variety of microscopy images, including plant cells, a lithium-ion battery cathode and organic crystals. The richness of these upsampled features admits separation of hard to segment phases, like hairline cracks. We demonstrate that interactive segmentation with these deep features produces high-quality segmentations far faster and with far fewer labels than training or finetuning a more traditional convolutional network.

Tables

Table 1. Table 1: Average mIoU of the three classes relative to the ground truth for three example benchmark datasets: Ni superalloys 7 , human T-cells 50 and Cu ore 51 - in both 4- and fully-labelled images. Standard deviation is measured across images in the dataset.

Dataset	$N_{\text{labels}}$	Class-avg mIoU
		Classical	+HR ViT
Ni	4	0.56 $\pm$ 0.16	0.75 $\pm$ 0.11
	22	0.73 $\pm$ 0.16	0.82 $\pm$ 0.10
T-cell	4	0.34 $\pm$ 0.09	0.64 $\pm$ 0.16
	22	0.41 $\pm$ 0.08	0.80 $\pm$ 0.13
Cu ore	4	0.85 $\pm$ 0.06	0.87 $\pm$ 0.03
	22	0.88 $\pm$ 0.05	0.89 $\pm$ 0.03

Table 2. Table S1: Training hyperparameters for the feature upsampler.

Input channels, $d_{in}$	$32/64/128/384$	Output channels, $d_{out}$	$32/64/128/384$
Hidden channels, $d_U$	$32/64/128/96$	Downsampler channels, $d_D$	$32/32/64/64$
Kernel size, $k$	3	Augmentations	flip_h, flip_v, rotate $(0,90,180,270)$
Normalization	L1	Loss	Smooth L1
Optimizer	AdamW	Learning rate	1e-4
Batch size	32	Epochs	5000

Table 3. Table S2: mIoU on the Ni superalloy dataset with 4 labelled examples as a function of number of added HR ViT features.

Number of added HR ViT channels	Class-avg mIoU
0	0.53 $\pm$ 0.14
1	0.59 $\pm$ 0.15
16	0.72 $\pm$ 0.16
32	0.73 $\pm$ 0.16
64	0.74$\pm$ 0.16
128	0.74 $\pm$ 0.16

Table 4. Table S3: Average mIoU of the three classes relative to the ground truth for Ni superalloy dataset with different feature-stacks concatenated to the classical feature-stack. We see that the uninformative or deleterious additions (uniform zeros, random noise etc. ) do not improve performance, whereas the HR ViT features do.

Case	Class-avg mIoU
Classical	0.56 $\pm$ 0.16
+Random noise	0.56 $\pm$ 0.15
+Uniform	0.56 $\pm$ 0.15
+Duplicated channels	0.56 $\pm$ 0.15
+HR ViT	0.75 $\pm$ 0.11

Table 5. Table S4: Average mIoU of the three classes relative to the ground truth for Ni superalloy dataset with different classifiers for the different feature-sets.

Classifier	Class-avg mIoU
	Classical	+HR ViT
Linear	0.40 $\pm$ 0.08	0.57 $\pm$ 0.12
Logistic	0.40 $\pm$ 0.08	0.63 $\pm$ 0.08
Random Forest	0.55 $\pm$ 0.16	0.68 $\pm$ 0.15
XGB	0.56 $\pm$ 0.15	0.74 $\pm$ 0.13
MLP	0.55 $\pm$ 0.11	0.69 $\pm$ 0.12

Full text

1 Introduction

Segmentation of micrographs is necessary for most kinds of image analysis in materials science, be it phase quantification 1, 2, 3, 4, 5, morphological characterization (particle size distribution 6 7, inter-particle distance 8 etc.) or transport simulations (e.g, tortuosity factor calculation 9, 10). However, it is often very challenging to segment these micrographs accurately: they can display imaging artefacts (low signal-to-noise ratio, pore-back3, charging and edge effects in electron microscopy; angular undersampling or beam hardening and photon starvation artefacts in computed tomography11), include complex or ambiguous phases (cracks, polymorphs), and are fundamentally physically distinct from natural images (i.e, typical photographs).

The gold standard for scientific image segmentation - the convolutional neural network (CNN) - often requires a large amount of (often densely) labelled training data 12, 13. This can be expensive and time consuming to collect, even with recent tools 4, 14, 15, and the subsequent models can take a long time to train or fail to generalise when applied to new, unseen systems or stoichiometries. Researchers often have to change their setups, and labelling, creating and training (or fine-tuning) a CNN each time represents a substantial challenge. Adaptable pretrained backbones, like Cellpose 16 and MicroNet 7, go some way to fixing this, though training a head network or finetuning still requires time and densely labelled training data. Classical methods like Otsu thresholding 17, k-means clustering 18 or watershed segmentation 19 require no ‘training data’ and little user input, but can be sensitive to noise 20, are limited in their expressive capacity, and often require multiple rounds of manual preprocessing to work well 21.

Trainable or interactive segmentation methods like Weka 22 or ilastik 23 extract classical features like average local intensities, edge magnitudes and textures, and train a classifier (usually a random forest) to map from those features to user-drawn pixel-labels. They can account for noise, exposure variation and train quickly, with only sparse labels needed. This makes them appealing in data scarce fields like materials science, where the imaging setup, phase of interest, processing conditions and stoichiometry can change frequently.

Despite their success, interactive segmentation with classical features can struggle with complex phases in microstructures 24. Various extensions exist, like ‘autocontext’ (retraining on the output classifier probabilities) 25 or enforcing experimental volume fractions in the loss 26, but these are still fundamentally limited by their original feature-set.

Feature foundation models like DINO 27, DINOv2 28 and I-JEPA 29 are models trained to learn general purpose image features over large datasets. They offer rich semantic features, but describe coarse ‘patches’ of an image (as opposed to pixel-level classical features). Recently work has been done on upsampling these features30, 31, 32, 24, 33, as well as on integrating these features into interactive segmentation workflows24, but practical considerations (memory/time scaling, feature quality) has limited success.

In this work, we introduce an efficient convolutional feature upsampler and leverage it for interactive segmentation of various multi-phase material systems: a lithium-ion cathode, nickel-superalloys and copper ore. The efficient design of the upsampler allows it to be used in an application context, in real-time, and on a laptop GPU. We summarise our contribution in Figure 1.

2 Background

2.1 Interactive feature-based segmentation

Interactive segmentation, first popularised in biological image analysis 22, 23, 34, trains a classifier to map from a features that describe a given pixel to the pixel’s class, over a set of user-defined labels. These features are computed over a range of length scales, $\boldsymbol{\sigma}$, often $\boldsymbol{\sigma}=\{\sigma_{\text{min}},...,2^{n-1}\sigma_{\text{min}}\}$ defining the size of a kernel applied to every pixel. They are ‘classical’ in that they are not learned, and reflect our understanding of information useful (and physically significant) for phase segmentation: average colour via Gaussian blurs, edge intensity via Sobel or Laplacian filters, texture via a Hessian filter, etc. 22, 23 The choice of features/filters used - the ‘feature-set‘ - produces a $(H,W,N)$ ‘feature-stack’ for a $(H,W)$ image.

Users draw labels onto pixels, indicating them as one of $C$ classes, e.g. grain/grain boundary, active material/pore/binder or nucleus/cytoplasm/organelles etc. A classifier $f_{\theta}$ is then trained to map from the $N$ dimensional features of labelled pixels to their class, $f_{\theta}:\mathbb{R}^{N}\rightarrow\mathbb{Z},\mathbb{Z}\in[1,...,C]$, and applied to the unlabelled pixels to produce a segmentation. This classifier is usually a random forest, which is capable of expressing nonlinear relationships not typically captured in the classical features22.

2.2 Vision transformers (ViTS)

A transformer is a deep neural network that uses ‘attention’, which is the dot-product similarity of learned projections of parts of its input, as its primary mechanism 36. An input is broken into $T\in\{t_{1},t_{2},...\}$ tokens, which are $k$ dimensional vectors and the network learns to update the representation of token $t_{i}$ with respect to all other tokens in $T$ to minimise some overall loss 36.

Transformers were initially applied to sequence modelling tasks, where the input tokens were (approximately) words in a sentence. The all-to-all nature of attention means the computational cost scales $\mathcal{O}(n_{t}^{2})$ where $n_{t}$ is the number of tokens 36. Therefore, to apply transformers to images, inputs had to be broken in non-overlapping patches - usually $16\times 16$ or $8\times 8\>\text{px}^{2}$ - as giving each pixel its own token would be computationally impractical 37.

The attention mechanism is fundamentally unordered (i.e, invariant under permutations of the input), so additional positional information has to be added to tokens via a positional encoding 36. This can be simple, like raster order 37, more complex like a sinusoidal encoding 38, or even learned during training.

Vision transformers have been applied to tasks like classification 37, depth perception 39 and segmentation 14, and, due to the favourable scaling of transformers with added data, have quickly reached state-of-the-art results.

2.3 Feature learning & DINO

Self-supervised learning (SSL) 40 is where a network learns representations by mapping between transformations or augmentations of an input, for example learning to fill in missing patches of an image 41 - the training target is simply the whole uncovered image. SSL is attractive because it admits learning without (expensive) human labels, and learns robust representations which can later be used in a supervised context via fine-tuning or adaptor networks 40.

DINO (self-distillation with no labels)27 used a student-teacher framework to learn semantic image features from local-global correspondences in a fully self-supervised framework. They found that this approach learnt (without explicit guidance) semantic decompositions of scenes, and that its features could be used for classification with a linear probe.

DINOv2 28 continued this approach at-scale, using more data and a series of techniques (new losses, regularisations, etc.) to stabilise the training process. The model’s features were semantically rich and consistent ‘despite changes of pose, style or even objects’28; they then trained a series of head networks (linear probes or convolutional heads for dense tasks) on top of those frozen features.

2.4 Feature upsampling

Features from ViTs are rich, but coarse, and in some domains (like materials science) there may not be enough labelled training data to train a convolutional head network for a given dense task (as in ref. 28). This motivates ‘feature upsampling’: increasing the resolution of these features up to full-pixel resolution or otherwise. These algorithms often use information from the image in order to perform the upsampling.

Various approaches to ViT feature upsampling have been attempted. One of the first was to reduce the stride of the patch projection layer (i.e, moving to overlapping patches): this worked without any further modification or training, but came with increased memory and time cost due to the increased number of tokens, as well as feature blurring 30, 24.

FeatUp 31 introduced two methods for feature upsampling: a fast, single-pass Joint Bilateral Upsampling filter (JBU) and a slower, per-image implicit approach. Both were trained by comparing features extracted by a ViT over different views/transformations (flips, zooms, pads) of an input image(s), drawing analogy to Neural Radiance Fields (NeRFs) 42. The JBU approach is fast, but has steep memory scaling and can suffer from blurring; the implicit approach has sharp features but must be trained separately for each image, which is slow.

LiFT 32 trains a feature upsampling network in a self-supervised fashion, by extracting features of a half-resolution image, upsampling them with a CNN and comparing to the (ground-truth) features of the full-resolution image. Similar to FeatUp implicit, they use a trained downsampler to extract salient features from the image for upsampling; the approach is lightweight in terms of time and memory.

LoftUp 33 learns a ‘coordinate-based feature upsampler’ (a coordinate transformer) and uses sharp SAM-based 14 ‘pseudo ground truths’ for training. This results in sharp upsampled feature maps and fast performance, but with large increases in memory cost as a function of image size, due to every pixel being a token and the quadratic scaling of attention.

3 Method

Our approach is to learn a convolutional upsampler network to map from an image and its corresponding low-resolution features as extracted by a ViT to the high-resolution map produced by training an implict FeatUp network on that image. We use a convolutional network for its attractive time and memory cost with respect to input image size, and for its general suitability to image tasks. A diagram of the method can be seen in Figure 2.

In this section we explain:

• •

How we trained our upsampler to increase the resolution of the coarse ViT features with respect to the image, including the model architecture and the process of generating a set of high-resolution ‘ground-truth‘ features.

• •

The preprocessing steps (PCA) for the input low-resolutions features required by our choice of ‘ground-truth‘ during training, and the deep feature visualisations throughout the paper.

• •

How these upsampled ViT features are then used in interactive feature-based segmentation (i.e, by concatenation with existing classical features).

3.1 Upsampler

Choice of backbone

Our approach works across a variety of ViT backbones, because it is trained on FeatUp implict ground-truths, which can be generated for various ViT and convolutional feature extractors 31. DINO models with register tokens perform better 43, as they have no anomalous background tokens with high attention, which can cause pathological behaviour in the FeatUp implict upsampler for some images. We experiment using a fine-tuned DINOv2 checkpoint trained on 3D geometry, FiT3D44, which has sharper feature-maps and therefore may make it easier for our network to learn to upsample.

Dataset

We train on a 2000 image subset of Imagenet Reduced, a set of 10,000 images taken from ImageNet 45, a general-purpose classification dataset of natural images. First, we generate low-resolution and high-resolution embedding pairs offline for each image by training a FeatUp implicit network offline. This is the longest step of the process, taking around three minutes per-image on a A100 GPU. Then, we train our upsampler to map from the image and its low-resolution embedding to the high-resolution ‘ground truth’.

Any generalist image dataset would work, and in future the upsampler could be trained on a larger dataset for better performance. A two-step training process could also improve performance (as in 28), where the network is initially trained on $256\times 256$ images and later on larger, $512\times 512$ images.

Architecture

Our model has two components: first a learned convolutional downsampler extracts features from input image $\bf{x_{i}}$ (of size $H\times W$) iteratively, halving the resolution each time. These image features are used as guidance for a convolutional upsampler which takes as input the low-resolution ViT features $\bf{x_{f}}$, of dimension $(H/p,W/p,k)$. $p$ is the original patch size of the ViT and $k$ is the dimenion of the PCA from the FeatUp preprocessing, usually $k=128$ (see Section 3.2).

$\bf{x_{f}}$ is upsampled $log_{2}(p)$ times until the features are at full resolution i.e, $(H,W,k)$, and then compared via smooth MSE loss to the FeatUp implict ‘ground truth’. The layers of the down- and upsampler are taken from standard U-Net implementations; we note that the network acts similar to a U-Net, with the addition of the LR feature concatenation at the bottleneck 12. Further network architecture and hyperparameter details are available in Section S2.

3.2 Feature preprocessing

When training their implicit upsampler, FeatUp prepares a dataset of features from a ViT of $N_{t}$ transforms of the original input image, then computes a shared PCA over these features (we call this ‘$\bf{x}_{\text{FU},\text{lr}}$’). This shared PCA is then used to project the features of the original, untransformed image. This is part of their NeRF analogy - ”that multiview consistency of low-resolution signals can supervise the construction of high-resolution signals”. Crucially, this shared PCA changes both the number of channels and the distribution of features when compared to the features produced by a ViT for the untransformed image in both low and high-resolution. This change is sometimes small when visualised (see Sections 3.2 and S3), but is often significant.

The full dataset for the implicit network consists of $3000$ transformations of the image, but a PCA over $N_{t}=50$ is a good approximation (i.e, the low- and high-res appear similar when visualised). We found difficulty in training an upsampler end-to-end when taking the original ViT features and upsampling them with respect to the FeatUp implicit features due to this distribution shift - see Section S3. We also attempted to train a ‘feature-transfer’ network to map from the original ViT features to $\bf{x}_{\text{FU},\text{lr}}$, but were unsuccessful.

To avoid this problem, we preprocess our low-resolution ViT features using this shared PCA before upsampling. This is done offline during training, alongside the high-resolution feature creation, and is computed and applied before upsampling at test-time. This means the output features from the model have implicitly had a PCA applied to them (the first feature channels capture the most variance), and that given a dataset of low- and high-resolution features of $k$ dimensions, compressed upsamplers of dimension $j<k$ can be trained by simply taking the first $j$ features, useful for GPU memory-constrained environments.

Feature Visualisation

As in previous works, we visualise our high dimensional features by taking a PCA over the features from $k=128$ or $d=384$ dimensions down to 3, and plotting the result as RGB channels, such that similar colours indicate similar features 28, 30. We do not need to train a PCA for FeatUp style features (see Section 3.2). This visualisation, whilst useful, does not explain the full picture, especially for classical features where there are more edge-based filters. This means edge features explain more variance and so are highlighted, but it does not mean colour information (from Gaussian blurs etc.) is absent, merely unemphasised.

3.3 Interactive segmentation workflow

Following a similar setup to previous work 22, 23, 24, we extract and concatenate our deep features to the classical ones before training a classifier to map to the user labels. We focus on classifiers which train quickly, like random-forests46, XGBoost47 and logistic regression models. This, combined with the efficient nature of our feature upsampler, allows quick, real-time interactive segmentation (see Section 4).

Explicitly, the classical process trains a pixelwise classifier $f_{\theta}:\mathbb{R}^{N}\rightarrow\mathbb{Z},\mathbb{Z}\in[1,...,C]$ over the set of features of all labelled pixels where $N$ is the dimension of the vector produced using the ‘classical’ image processing operations and $C$ is the number of classes. We concatenate the $k$-dimensional (upsampled) deep features for each pixel onto the $N$-dimensional classical features and train a classifier $f^{\prime}_{\theta}$, $f^{\prime}_{\theta}:\mathbb{R}^{N+k}\rightarrow\mathbb{Z},\mathbb{Z}\in[1,...,C]$ - we denote this as ‘+ HR ViT features’ or ‘+HR ViT’.

‘Classical’ refers to a Python reimplementation of the Weka featurisation process described in Section 2.1, which we use as a baseline comparison. The same labels and classifier type were used for all ‘Classical’ and ‘+ HR ViT features’ comparisons. Unless otherwise stated, we use our convolutional upsampler to upsample DINOv2-S features following Sections 3.1 & 3.2, and an XGBoost classifier (with default hyperparameters) to map from features to labels.

We note no pre-processing (denoising, histogram equalization, etc) or post-processing (e.g, modal filters, small hole removal, etc) were used, though the approach is of course compatible with them.

3.4 Evaluation metric

For evaluating segmentation performance in Section 4 we use mean Intersection over Union (mIoU). The mIoU (for a binary segmentation) measures the ratio of true positives to true positives plus false positives (i.e, the positive predictive value) across all the pixels in a prediction when compared to a ground truth. This can then be applied to multi-class segmentation by treating the predictions for each class as a binary segmentation and averaging over them.

4 Results & discussion

Performance landscape

To be used successfully in interactive segmentation, it must be quick and cheap to compute the high-resolution deep features, and these features should be well-localised/sharp. Therefore, we compare the time and memory costs of the various generalised upsampling approaches (strided, JBU, LiFT, LoftUp, ours) in Figure 3, plotting them alongside an example feature-map for a reference image. More qualitative examples of feature map comparisons for micrographs can be found in Section S4.

We find that our approach and LiFT have the most favourable time & memory scaling (strided has high time cost; JBU and LoftUp a high memory cost), and similar feature sharpness. Our approach has more focus on semantics rather than colour information, but this is because it is trained on DINOv2 whereas LiFT is trained on DINO.

The ability to train a ‘compressed’ upsampler for $k=16$ features (see Section 3.2) allows us to process relatively large ($\sim 2000\times 1000$ pixels) images on a laptop GPU (4GB NVIDIA RTX 3050M), as the cost of storing a $(H,W,64/128/384)$ in memory or on disk scales rapidly. This improved scaling, and the efficiency of our approach in general, is important for microscopists, whose images can often be hundreds of megapixels.

4.1 Interactive segmentation

Following our previous work24, we concatenate these upsampled deep features with the classical features to perform interactive segmentation for a variety of material micrographs. We do this for large micrographs displaying complex phases, as well as testing their generalisation ability on predefined benchmark datasets.

In both the examples and the benchmarks we find that adding the HR ViT features offers the classifier additional relevant information on which it can condition its predictions - we can see evidence of this in the feature map visualisations in Figure 1 and in Section S6. Despite not being trained on micrographs, the ViT models have learned useful primitives that generalise, like foreground vs background, interiority, (perceived) depth, shape etc.

Single-image examples

Figure 4 shows a comparison between the segmentations produced using the ‘Classical’ features and the ‘+ HR ViT features’ for a series of scanning electron microcopy (SEM) micrographs. The first is an SEM of a battery cathode with four phases48: pore (and out-of-plane material), carbon binder, and two different active materials (light and dark grey particles). The ViT features allow a much better distinction between material in (i.e, flat, usually curtained) and behind the imaging plane. This distinction is important for accurately determining a battery’s porosity for use in later physics-based simulations3, and for understanding the impact of processing conditions like calendaring.

The second micrograph is dual-phase steel, with ferrite, martensite (raised) and some damage (black voids)48. We see that the classical features struggle to consistently distinguish the centres of large martensite regions and the flat ferrite regions. The third is an SEM of plant cells49, where the four phases are the background, cell wall, cytoplasm and nucleus; the classical features struggle to differentiate between the nucleus and the rest of the cells due to similar intensities.

Each of these images was roughly $2000\times 1000$ pixels large, and was segmented on a laptop GPU on a custom interactive segmentation app. Although our upsampler is most suited to this specific environment, we note that the other upsamplers could be used, were more memory available (FeatUp JBU, LiFT). We study the impact of the ViT feature upsampler on interactive segmentation in Section S4, finding that the simple approaches (bilinear upsampling, strided) cause artefacts.

On benchmark datasets

Few labelled interactive segmentation datasets exist for materials science, and there are fewer ‘benchmark datasets’ which include examples across different material systems and instruments. We collate a series of three datasets of micrographs and their associated ground truth segmentations into a modest benchmark which aims to test the accuracy and generalizability of the feature-sets.

The first dataset, ‘Ni’, is a set of SEM micrographs of nickel superalloys with tertiary precipitates, taken from ref 7. The second, ‘T-cell’, is series of transmission electron microscopy (TEM) images of Jurkat T-cells 50 with three classes: background, nucleus, cytoplasm. The third dataset ‘Cu ore’ is reflected light microscopy (RLM) images of copper ore in epoxy resin 51, with two classes: ore and resin (background).

For each of these datasets we train using the features and labels of four exemplar images and apply the classifiers unseen to the other images in the dataset (18, 26 and 26 respectively). We measure the mIoU of the prediction relative to the ground truth, and report the results in Table 1. Example train and test data, as well as classifier predictions, are available in Figure 5. The mIoU is measured across all images (train included) - this is justified as the labels are themselves sparse (i.e, do not cover the whole image) and are consistent across feature-sets. Labels were produced using the SAMBA web-app’s smart labelling tool15.

In agreement with previous observations, we see that the ViT features allow more accurate segmentations, increasing the mIoU (sometimes greatly) across all datasets. For the Ni superalloys, this means more consistency across large exposure variations, especially of the tertiary precipitates. On the T-cells this manifests as better distinction between foreground/background cells, and of the interior nucleus.

Finally, we study the long limit on these benchmark datasets, increasing the number of sparsely-labelled images we train the classifiers over, and measuring the mIoU over all images each time. We present the results in Figure 6 (and the highest achieved value in Table 1), and see that the deep features scale well with added training data, achieving high mIoUs.

Some sharp increases in performance for the classical feature-set can be seen, this is interpretable as a data-ordering problem, where some useful feature or ambiguity becomes labelled and performance across the rest of the images increases. This ordering however does not affect the result in the long limit.

Example predictions for these ‘fully-trained’ classifiers on the Ni superalloys is available in Figure S8, we note that the predictions with +HR ViT features align much more closely with the ground truth. For some datasets (Ni, T-Cell) it took five to six times longer to train classifiers on the classical features than the +HR ViT features - the trees had to be grown to a larger depth to make adequate splits. This time increase was not seen for the Cu ore dataset, where the classical and +HR ViT feature-sets had more similar performance.

It is unsurprising that the addition of the HR ViT features improves segmentation performance - all the classical feature-set information is still present - but the increase in perceived quality and mIoU is marked, as is the speed increase during classifier training. We demonstrate this improvement is not attributable to regularization (i.e, by comparing to the addition of random or uninformative channels) in Section S7 - and find the +HR ViT features must be supplying new, useful information to the classifier.

Explicit comparison to CNNs

In order to justify our claim that ‘you may not need a U-net’, we compare our approach to a recent micrograph-pretrained CNN model, ‘MicroNet’ 7. MicroNet is a U-Net architecture with a series of backbones that have been trained on ImageNet 45 and then on a proprietary large micrograph dataset. It is then fine-tuned using task-specific image and segmentation mask pairs - we view this large backbone + task-specific training as similar to the (deep) feature extraction + classifier training of our approach.

We compare the segmentation quality (mIoU) over all 22 images as a function of time for MicroNet and our approach in Figure 7. Both approaches were trained using four micrograph - segmentation pairs from the Ni-superalloy dataset introduced by MicroNet. We either train on the sparse labels from Section 4 ($\sim 20\%$ pixels labelled), or using the full ground truth segmentations. For MicroNet time is measured over each epoch of training, and for HR ViT time is measured for featurisation + classifier training + classifier application for all 22 images (i.e, a conservative estimate for our approach).

To train MicroNet on sparse labels we mask its standard Binary Cross Entropy loss to only the labelled pixels. Other parameters are kept the same (learning rate, number of epochs etc.), consistent across the different labelling methods (full vs. sparse). The U-Net++ configuration with a ResNet50 encoder 52 was used for MicroNet.

Our approach reaches a high mIoU quickly, even with sparse labels (75% in 20.4s for sparse labels, 79% in 38s for full labels). We note this was achieved using CPU XGBoost and a 128-channel upsampler - there is room for further speedup. MicroNet takes more time, taking around 4 minutes to reach 80% mIoU with full labels, and struggles to train well with sparse labels. This indicates our approach is markedly more time and data efficient than using a CNN, useful if fully-labelled ground truths are unavailable or costly to obtain (fully-labelling large micrographs can take days53).

However, given longer time we expect MicroNet (with fully-labelled data) would outperform HR ViT, which we observe in Figure 7. This is not unexpected: fine-tuning feature extraction on task-specific data should lead to the best performance.

We stress that the featurisation step for +HR ViT need only be computed once, and further labels can be added to quickly retrain and improve the classifier, for example on other images in the validation set. This compounds the advantage over the CNN: mistakes can be corrected interactively using sparse labels well before the CNN has finished training.

5 Conclusions & future work

To conclude, we have introduced a fast convolutional feature upsampler for ViT features and demonstrated the capacity of these features to greatly improve interactive micrograph segmentation. We do this for a range of materials, instruments and image resolutions, in resource constrained environments all in matter of seconds. This approach is flexible, outperforms CNNs, and requires far less labelled training data.

There is scope for future work to improve various aspects of the approach: reducing or removing the positional bias implicit in the ViT features (see Section S11), finding ways of improving the feature-preprocessing step and ensuring consistency of ViT features (i.e, in a volumetric stack). It is also important to account for the fact that the majority of micrographs are greyscale, and that feature foundation models (and our upsampler) are trained on natural, RGB images. Training a feature foundation model on micrographs could ameliorate this, as well as allowing better expressivity of microstructural features likes grains, cracks and pores.

We hope this work will encourage further integration (and clever adaptation) of foundation model features into materials imaging workflows, like segmentation, denoising or property prediction.

Code Availability

The Python reimplementation of the classical featurisation process is available here and the GUI for the user app is available here. The code for the ViT feature upsampler and to reproduce the paper figures, alongside instructions on how to install these three packages together, is available here.

Data Availability

The data used in this study (micrographs, labels) are available on zenodo.

Acknowledgements

This work was supported by funding from the the EPRSC and SFI Centre for Doctoral Training in Advanced Characterisation of Materials (EP/S023259/1 received by RD) and the Royal Society (IF\R2\222059 received by AV as a Royal Society Industry Fellow).

The authors would like to thank other members of the TLDR group for their testing and feedback.

Authorship

RD conceived the concept, wrote the code and performed analysis for the manuscript. AV and SJC contributed to the development of the concepts presented in all sections of this work and made substantial revisions and edits to all sections of the draft manuscript.

Competing interests

The authors declare no competing interests.

• 1

Z. Su, E. Decencière, T.-T. Nguyen, K. El-Amiry, V. De Andrade, A. A. Franco, and A. Demortière, “Artificial neural network approach for multiphase segmentation of battery electrode nano-CT images,” npj Computational Materials, vol. 8, p. 30, Feb. 2022.

• 2

S. Müller, P. Pietsch, B.-E. Brandt, P. Baade, V. De Andrade, F. De Carlo, and V. Wood, “Quantification and modeling of mechanical degradation in lithium-ion batteries based on nanoscale imaging,” Nature Communications, vol. 9, p. 2340, June 2018.

• 3

S. J. Cooper, S. A. Roberts, Z. Liu, and B. Winiarski, “Methods—Kintsugi Imaging of Battery Electrodes: Distinguishing Pores from the Carbon Binder Domain using Pt Deposition,” Journal of The Electrochemical Society, vol. 169, p. 070512, July 2022.

• 4

L. H. Rieger, F. Cadiou, Q. Jacquet, V. Vanpeene, J. Villanova, S. Lyonnard, T. Vegge, and A. Bhowmik, “Utilizing active learning to accelerate segmentation of microstructures with tiny annotation budgets,” Energy Storage Materials, vol. 73, p. 103785, 2024.

• 5

S. Medghalchi, J. Kortmann, S.-H. Lee, E. Karimi, U. Kerzel, and S. Korte-Kerzel, “Automated segmentation of large image datasets using artificial intelligence for microstructure characterisation and damage analysis,” Materials & Design, vol. 243, p. 113031, 2024.

• 6

H. Xu, J. Zhu, D. P. Finegan, H. Zhao, X. Lu, W. Li, N. Hoffman, A. Bertei, P. Shearing, and M. Z. Bazant, “Guiding the Design of Heterogeneous Electrode Microstructures for Li-Ion Batteries: Microscopic Imaging, Predictive Modeling, and Machine Learning,” Advanced Energy Materials, vol. 11, p. 2003908, May 2021.

Publisher: John Wiley & Sons, Ltd.

• 7

J. Stuckner, B. Harder, and T. M. Smith, “Microstructure segmentation with deep learning encoders pre-trained on a large microscopy dataset,” npj Computational Materials, vol. 8, p. 200, Sept. 2022.

• 8

S. Müller, P. Pietsch, B.-E. Brandt, P. Baade, V. De Andrade, F. De Carlo, and V. Wood, “Quantification and modeling of mechanical degradation in lithium-ion batteries based on nanoscale imaging,” Nature Communications, vol. 9, p. 2340, June 2018.

• 9

S. Cooper, A. Bertei, P. Shearing, J. Kilner, and N. Brandon, “TauFactor: An open-source application for calculating tortuosity factors from tomographic data,” SoftwareX, vol. 5, pp. 203–210, 2016.

• 10

S. Kench, I. Squires, and S. Cooper, “Taufactor 2: A gpu accelerated python tool for microstructural analysis,” Journal of Open Source Software, vol. 8, no. 88, p. 5358, 2023.

• 11

P. J. Withers, C. Bouman, S. Carmignato, V. Cnudde, D. Grimaldi, C. K. Hagen, E. Maire, M. Manley, A. Du Plessis, and S. R. Stock, “X-ray computed tomography,” Nature Reviews Methods Primers, vol. 1, p. 18, Feb. 2021.

• 12

O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation,” 2015.

• 13

S. Reiß, C. Seibold, A. Freytag, E. Rodner, and R. Stiefelhagen, “Graph-constrained Contrastive Regularization for Semi-weakly Volumetric Segmentation,”

• 14

A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. Girshick, “Segment anything,” arXiv preprint, arXiv:2304.02643, 2023.

• 15

R. Docherty, I. Squires, A. Vamvakeros, and S. J. Cooper, “SAMBA: A Trainable Segmentation Web-App with Smart Labelling,” Journal of Open Source Software, vol. 9, no. 98, p. 6159, 2024.

Publisher: The Open Journal.

• 16

M. Pachitariu and C. Stringer, “Cellpose 2.0: how to train your own model,” Nature Methods, Nov. 2022.

• 17

N. Otsu, “A Threshold Selection Method from Gray-Level Histograms,” IEEE Transactions on Systems, Man, and Cybernetics, vol. 9, no. 1, pp. 62–66, 1979.

• 18

J. A. Hartigan and M. A. Wong, “Algorithm as 136: A k-means clustering algorithm,” Journal of the Royal Statistical Society. Series C (Applied Statistics), vol. 28, no. 1, pp. 100–108, 1979.

• 19

F. Meyer, “Topographic distance and watershed lines,” Signal Processing, vol. 38, no. 1, pp. 113–125, 1994.

• 20

J. T. Barron, “A Generalization of Otsu’s Method and Minimum Error Thresholding,” arXiv preprint, arXiv:2007.07350, 2020.

• 21

F. L. E. Usseglio-Viretta, A. Colclasure, A. N. Mistry, K. P. Y. Claver, F. Pouraghajan, D. P. Finegan, T. M. M. Heenan, D. Abraham, P. P. Mukherjee, D. Wheeler, P. Shearing, S. J. Cooper, and K. Smith, “Resolving the Discrepancy in Tortuosity Factor Estimation for Li-Ion Battery Electrodes through Micro-Macro Modeling and Experiment,” Journal of The Electrochemical Society, vol. 165, p. A3403, nov 2018.

Publisher: The Electrochemical Society.

• 22

I. Arganda-Carreras, V. Kaynig, C. Rueden, K. W. Eliceiri, J. Schindelin, A. Cardona, and H. Sebastian Seung, “Trainable Weka Segmentation: a machine learning tool for microscopy pixel classification,” Bioinformatics, vol. 33, pp. 2424–2426, Aug. 2017.

• 23

S. Berg, D. Kutra, T. Kroeger, C. N. Straehle, B. X. Kausler, C. Haubold, M. Schiegg, J. Ales, T. Beier, M. Rudy, K. Eren, J. I. Cervantes, B. Xu, F. Beuttenmueller, A. Wolny, C. Zhang, U. Koethe, F. A. Hamprecht, and A. Kreshuk, “ilastik: interactive machine learning for (bio)image analysis,” Nature Methods, Sept. 2019.

• 24

R. Docherty, A. Vamvakeros, and S. J. Cooper, “Upsampling dinov2 features for unsupervised vision tasks and weakly supervised materials segmentation,” arXiv preprint, arXiv:2410.19836, 2024.

• 25

X. Bai and Z. Tu, “Auto-Context and Its Application to High-Level Vision Tasks and 3D Brain Image Segmentation,” IEEE Transactions on Pattern Analysis & Machine Intelligence, vol. 32, pp. 1744–1757, Oct. 2010.

Place: Los Alamitos, CA, USA Publisher: IEEE Computer Society.

• 26

N. Prakash, P. Gasper, and F. Usseglio-Viretta, “ExpertSegmentation: Segmentation for microscopy with domain-informed targets via custom loss,” Acta Materialia, vol. 291, p. 120993, 2025.

• 27

M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging Properties in Self-Supervised Vision Transformers,” arXiv preprint, arXiv:2104.14294, 2021.

• 28

M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P.-Y. Huang, S.-W. Li, I. Misra, M. Rabbat, V. Sharma, G. Synnaeve, H. Xu, H. Jegou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski, “DINOv2: Learning Robust Visual Features without Supervision,” arXiv preprint, arXiv:2304.07193, 2023.

• 29

M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. Rabbat, Y. LeCun, and N. Ballas, “Self-supervised learning from images with a joint-embedding predictive architecture,” arXiv preprint, arXiv:2301.08243, 2023.

• 30

S. Amir, Y. Gandelsman, S. Bagon, and T. Dekel, “Deep ViT Features as Dense Visual Descriptors,” arXiv preprint, arXiv:2112.05814, 2022.

• 31

S. Fu, M. Hamilton, L. Brandt, A. Feldman, Z. Zhang, and W. T. Freeman, “FeatUp: A Model-Agnostic Framework for Features at Any Resolution,” arXiv preprint, arXiv:2403.10516, 2024.

• 32

S. Suri, M. Walmer, K. Gupta, and A. Shrivastava, “LiFT: A Surprisingly Simple Lightweight Feature Transform for Dense ViT Descriptors,” arXiv preprint, arXiv:2403.14625, 2024.

• 33

H. Huang, A. Chen, V. Havrylov, A. Geiger, and D. Zhang, “LoftUp: Learning a Coordinate-Based Feature Upsampler for Vision Foundation Models,” arXiv preprint, arXiv:2504.14032, 2025.

• 34

R. Haase, D. Lee, D. D. Pop, and L. Žigutytė, “haesleinhuepf/napari-accelerated-pixel-and-object- classification: 0.14.1,” Nov. 2023.

• 35

T. D. Turner, P. Gajjar, I. S. Fragkopoulos, J. Carr, T. T. H. Nguyen, D. Hooper, F. Clarke, N. Dawson, P. J. Withers, and K. J. Roberts, “Measuring the Particle Packing of l-Glutamic Acid Crystals through X-ray Computed Tomography for Understanding Powder Flow and Consolidation Behavior,” arXiv preprint, arXiv:https://doi.org/10.1021/acs.cgd.9b01515, vol. 20, no. 7, pp. 4252–4263, 2020.

• 36

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017.

Type: Journal Article.

• 37

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” arXiv preprint, arXiv:2010.11929, 2021.

• 38

M. Tancik, P. P. Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan, U. Singhal, R. Ramamoorthi, J. T. Barron, and R. Ng, “Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains,” arXiv preprint, arXiv:2006.10739, 2020.

• 39

L. Yang, B. Kang, Z. Huang, X. Xu, J. Feng, and H. Zhao, “Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data,” arXiv preprint, arXiv:2401.10891, 2024.

• 40

R. Balestriero, M. Ibrahim, V. Sobal, A. Morcos, S. Shekhar, T. Goldstein, F. Bordes, A. Bardes, G. Mialon, Y. Tian, A. Schwarzschild, A. G. Wilson, J. Geiping, Q. Garrido, P. Fernandez, A. Bar, H. Pirsiavash, Y. LeCun, and M. Goldblum, “A cookbook of self-supervised learning,” arXiv preprint, arXiv:2304.12210, 2023.

• 41

K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked Autoencoders Are Scalable Vision Learners,” arXiv preprint, arXiv:2111.06377, 2021.

• 42

B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis,” arXiv preprint, arXiv:2003.08934, 2020.

• 43

T. Darcet, M. Oquab, J. Mairal, and P. Bojanowski, “Vision Transformers Need Registers,” arXiv preprint, arXiv:2309.16588, 2023.

• 44

Y. Yue, A. Das, F. Engelmann, S. Tang, and J. E. Lenssen, “Improving 2d feature representations by 3d-aware fine-tuning,” arXiv preprint, arXiv:2407.20229, 2024.

• 45

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “Imagenet large scale visual recognition challenge,” arXiv preprint, arXiv:1409.0575, 2015.

• 46

L. Breiman, “Random Forests,” Machine Learning, vol. 45, pp. 5–32, Oct. 2001.

• 47

T. Chen and C. Guestrin, “Xgboost: A scalable tree boosting system,” in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, p. 785–794, ACM, Aug. 2016.

• 48

“Biphase cathode sem.”

https://forum.image.sc/t/three-phase-segmentation-of-sem-image/92812.

• 49

B. D. University of Wisconsin Stevens Point, “Electron micrographs.”

https://www4.uwsp.edu/biology/courses/botlab/Lab03c.htm.

• 50

V. Morath, M. Keuper, M. Rodriguez-Franco, S. Deswal, G. Fiala, B. Blumenthal, D. Kaschek, J. Timmer, G. Neuhaus, S. Ehl, O. Ronneberger, and W. W. A. Schamel, “Semi-automatic determination of cell surface areas used in systems biology.,” Frontiers in bioscience (Elite edition), vol. 5, pp. 533–545, Jan. 2013.

• 51

M. P. Filippo, O. d. F. M. Gomes, G. A. O. P. d. Costa, and G. L. A. Mota, “Deep learning semantic segmentation of opaque and non-opaque minerals from epoxy resin in reflected light microscopy images,” Minerals Engineering, vol. 170, p. 107007, 2021.

• 52

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” arXiv preprint, arXiv:1512.03385, 2015.

• 53

L. H. Rieger, F. Cadiou, Q. Jacquet, V. Vanpeene, J. Villanova, S. Lyonnard, T. Vegge, and A. Bhowmik, “Utilizing active learning to accelerate segmentation of microstructures with tiny annotation budgets,” Energy Storage Materials, vol. 73, p. 103785, 2024.

• 54

C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, M. Brett, A. Haldane, J. F. del Río, M. Wiebe, P. Peterson, P. Gérard-Marchant, K. Sheppard, T. Reddy, W. Weckesser, H. Abbasi, C. Gohlke, and T. E. Oliphant, “Array programming with NumPy,” Nature, vol. 585, pp. 357–362, Sept. 2020.

• 55

S. van der Walt, J. L. Schönberger, J. Nunez-Iglesias, F. Boulogne, J. D. Warner, N. Yager, E. Gouillart, and T. Yu, “scikit-image: image processing in python,” PeerJ, vol. 2, p. e453, June 2014.

• 56

M. Popeil, F. L. Usseglio-Viretta, X. Pu, P. Gasper, N. Dutta, E. Wang, E. Allen, J. S. Mangum, N. Sunderlin, K. Fink, J. M. Allen, P. J. Weddle, S. C. DeCaluwe, and D. P. Finegan, “Heterogeneity of the Dominant Causes of Performance Loss in End-of-Life Cathodes and Their Consequences for Direct Recycling,” Advanced Energy Materials, vol. n/a, no. n/a, p. 2405371.

• 57

V. Havrylov, H. Huang, D. Zhang, and A. Geiger, “Benchmarking feature upsampling methods for vision foundation models using interactive segmentation,” arXiv preprint, arXiv:2505.02075, 2025.

• 58

O. Siméoni, H. V. Vo, M. Seitzer, F. Baldassarre, M. Oquab, C. Jose, V. Khalidov, M. Szafraniec, S. Yi, M. Ramamonjisoa, F. Massa, D. Haziza, L. Wehrstedt, J. Wang, T. Darcet, T. Moutakanni, L. Sentana, C. Roberts, A. Vedaldi, J. Tolan, J. Brandt, C. Couprie, J. Mairal, H. Jégou, P. Labatut, and P. Bojanowski, “Dinov3,” 2025.

• 59

J. Eliot, “Doitpoms micrograph library,” 2000.

https://www.doitpoms.ac.uk/index.php.

• 60

R. F. Cochrane, “Micrograph 394,” 2002.

https://www.doitpoms.ac.uk/miclib/micrograph_record.php?id=394.

• 61

D. Cupid, “Mosilib: Innovative anode materials for more powerful and sustainable batteries.”

https://www.ait.ac.at/en/themen/battery-materials-characterization/projects/mosilib.

• 62

J. Yang, K. Z. Luo, J. Li, C. Deng, L. Guibas, D. Krishnan, K. Q. Weinberger, Y. Tian, and Y. Wang, “Denoising vision transformers,” arXiv preprint, arXiv:2401.02957, 2024.

Supplementary

S1 Classical feature-set

We used a Python reimplementation (using numpy54 and scikit-image55) of the default Weka22 feature-set for the classical features.

Gaussian blurs at each pixel for a set of strengths $\sigma\in\{0,1,2,4,8,16\}$. 2. 2.

Sobel edge detection at each pixel for each of the Gaussian filtered arrays in 1. 3. 3.

Hessian texture filter at each pixel for each of the Gaussian filtered arrays in 1, extracting the first two eigenvalues, as well as the mod, trace and determinant of the Hessian matrix. 4. 4.

Difference of Gaussians of each of the Gaussian arrays in 1. 5. 5.

Membrane projections: convolution of the image array with a stack of line kernels oriented at $30\degree$ angle increments in $[0\degree,180\degree).$

S2 Hyperparameters

The training hyperparameters for our upsampler is detailed in Table S1. The network architecture is shown in Figure S1.

S3 Feature preprocessing explained

FeatUp computes and applies a $k$-component PCA during training of their per-image implicit upsampler which we use as ground-truth high-resolution training targets. This PCA is computed over the features of 50 transformations of an input image. These transformations are randomly sampled and consist of flips, pads and zooms. This PCA not only changes the number of channels available in the feature-stack, but also shifts the feature distribution.

We present an example of this distribution shift in Figure S2 a., comparing the 128-component PCA of the untransformed image to subsequent PCAs with features of random transformations of the image included (i.e, the FeatUp PCA process). The distribution shift can be seen in the colour changes of the patch features. In Figure S2 b. we show the average patchwise MAE from the N-image PCA to the 1-image PCA, averaged over 100 images.

This data distribution shift made it more difficult to learn a lightweight upsampler from DINOv2 features to high-resolution FeatUp ground truths (which were implicitly PCA’d), as it had to learn both a complex distribution shift and the upsampling task. To fix this, we applied this feature preprocessing (i.e, projection with a PCA fit over the features of 50 transformations of the input image) to the low-resolution DINOv2 features before upsampling. We must therefore apply this preprocessing during inference on new images.

S4 Impact of upsampler on interactive segmentation

We compare different DINOv2 feature upsamplers when used in interactive segmentation in Figure S3. For the same image (XCT slice of a cracked NMC cathode56) and labels, we compute the classical features and concatenate them with the deep features from a given upsampler, train an XGB classifier to map features to labels and then apply to generate a segmentation.

We see the simple upsampling choices like nearest-neighbour upsampling and bilinear suffer from artefacting. Other approaches like FeatUp and Lift display either blurring or positional effects.

Similar to 24, 57, we find that bilinear feature upsampling is cheap and effective, especially when we combine it with classical features in interactive segmentation. The benefit of more complex upsamplers (ours, FeatUp, LoftUp, etc.) comes when the deep features are needed to distinguish fine features (less than the patch size) like cracks inside or the edges of particles in Figure S3.

This becomes more important as the patch size increases from 14px 27 to 16px in DINOv3 58. A side benefit of our approach is that compressed feature stacks ($k=16,32,64,128$) are easier to store (in memory or on disk) than the full $k=384$ from bilinear upsampling.

S5 How many deep features do we need?

S6 More feature visualisation examples

We present more upsampled DINOv2 feature visualisations for micrographs in Figure S4, noting that the model tends to display intuitive phase decompositions, even for phases with similar greyscale values. For example, the induced graphite in the iron alloy is similar in colour to the iron, but the DINOv2 features identify it as a distinct phase (possibly due to its interiority). The same can be seen in the carbon binder vs. graphite particles in the anode, which have the same greyscale values but are structurally distinct.

S7 Empirical justification of improved performance

Our +HR ViT approach appends $k$ channels (usually $k=32$) of upsampled, implicitly PCA’d (see Section 3.2) ViT features to an existing classical feature-stack. There is the possibility that the improvement in segmentation comes from having additional (redudant) channels to fit over, rather than these channels adding useful information i.e they act as regularization. Although it is unlikely for these extra channels to act as a regularization, it is possible if the classifier can only see a subset of the features at any given time, which is the case for the ‘random forest with feature sampling at splits’ formulation of Weka Trainable segmentation.

We test three cases: adding $k=32$ channels of i) random noise, ii) uniform zeros and iii) duplicating the first $k$ channels of the classical feature-stack, and measured segmentation performance on the Ni superalloy benchmark dataset. We report the results in Table S3, and find that the each baseline case fails to explain the performance increase of +HR ViT.

Our argument for this improvement is that these features contain additional useful information about the image that maps well to various phase/class distinctions, and allows a classifier to distinguish based on these. We see evidence of this in the feature map visualisations of Figure 1 and in Section S6. For example, imagine a classical setup trying to distinguish cracks inside a particle from narrow gaps between particles - their average colour on various length-scales may be similar, or even identical, and this may make a classifier struggle, and even degrade performance on other pixels.

If, on the other hand, the HR ViT features contain a notion of ‘particleness’, which is high inside a particle and low around, even in narrow gaps, the classifier can then learn to detect an internal crack using average colour and ‘particleness’, without degrading performance elsewhere. Of course, if there was spurious information in these ViT features the classifier could learn to split on these instead of generalising - we can see an example of such a case in Section S11.

S8 Ablation: choice of classifier

We experiment with changing the classifier used to map from feature-set $\rightarrow$ labels in Figure S5. Tree based methods (XGB and random forest) outperform (generalized) linear models like linear regression and logistic regression. These tree-based methods can model non-linear relationships than linear models cannot (this was the motivation for their use in Trainable Weka Segmentation)22.

Interestingly, we note that the +HR ViT feature-set can still segment the tertiary precipitates even with a linear model (when the classical features fail) - for this feature-set the non-linearity is implicit in the features, and does not need to be modelled with a non-linear classifier.

We repeat the benchmark experiment of Section 4 (4 labelled images, apply to 22, measure mIoU) for the Ni superalloy dataset with a variety of classifiers. XGB and Random Forest perform the best across feature-sets - we use XGB for its faster training speed. We note the relatively strong performance of the linear classifier + HR ViT features setup when compared to the classical feature-set.

S9 Further interactive segmentation examples

S9.1 On natural images

Figure S6 shows our approach (feature upsampling + interactive segmentation) applied to a set of natural images (i.e, not taken with a microscope). We see good upsamplings of low-res DINOv2 features, this is expected as our upsampler is trained on similar images. We also experiment with performing interactive segmentation on these images, comparing the results when using only the classical feature-set vs the combination of classical and upsampled DINOv2 features. Again we see much improved segmentations as the deep features express strong semantic decompositions.

S9.2 On micrographs

We show further examples of our approach improving micrograph segmentation in Figure S7. The active material (blue) and carbon binder (red) distinction is greatly improved for a tin-sulfide anode 61, as well as improved out-of-plane / in-pore differentiation. This ability to distinguish between in-plane (flat) and out-of-plane (textured) material is important for resolving the ‘pore-back’ problem in imaging of porous media, we see a further example of this when applied to an NMC cathode SEM3 in the second row. The classical features struggle to consistently differentiate between in-plane active material (blue) and out-of-plane material (grey), especially when compared to the carbon binder (yellow).

This improvement continues for a relfected light micrograph of cast iron alloy 60, which contains spherodised graphite (red). The classical features struggle to separate the graphite from the iron matrix (blue), due to a similar (average) colour - the DINOv2 features are able to capture this distinction (see Section S6 for an explanation).

S10 Predictions of fully trained classifiers

Figure S8 shows predictions of classifiers trained on labels across all 22 images in the dataset with either the Classical or +HR ViT dataset. Each image is shown with its labels (similar labels were applied to other unshown images) and the associated ground truth. The same labels were used for both feature-sets. We see that the +HR ViT features produce much better segmentations, with fewer mispredictions across each classes. Both approaches get the gross structure correct, but the noisiness of the Classical feature-set’s predictions make them unsuitable for counting precipitate size, or using in downstream simulations.

S11 Limitations: positional bias

ViTs have a positional encoding (see Section 2.2) added to each patch token which represents where that patch has come from in the image. This allows ViTs to express features over objects that span multiple patches, but also leads to this information being persisted into the resulting feature-stacks. This is undesirable, but can compensated for by labelling homogenously or by averaging over various transformations (flips etc.) 24. Work has been done on correcting similar artefacts in vision transformers 62.

We show the issues this causes for interactive segmentation in Figure S9. There is a clear gradient in the PCA visualisation of the HR ViT feature-stack, which results in a poor segmentation when a classifier is trained on only the HR ViT features (note: this is not the same case as +HR ViT, which refers to concatenating the ViT features to the classical ones). The homogenous classical features do not struggle, even with biased labels. Further discussion, alongside an example of using flip transforms to ameliorate this problem, can be found in ref. 24.