Mitigating Pretraining-Induced Attention Asymmetry in 2D+ Electron Microscopy Image Segmentation

TL;DR

This paper identifies and addresses the bias introduced by pretrained RGB models in electron microscopy segmentation, proposing a weight modification method to restore symmetry and improve interpretability without sacrificing accuracy.

Contribution

It reveals the channel asymmetry bias in pretrained models for electron microscopy and introduces a weight initialization technique to mitigate this bias.

Findings

Reduced attribution bias in models after weight modification

Maintained or improved segmentation accuracy

Enhanced model interpretability in electron microscopy tasks

Abstract

Vision models pretrained on large-scale RGB natural image datasets are widely reused for electron microscopy image segmentation. In electron microscopy, volumetric data are acquired as serial sections and processed as stacks of adjacent grayscale slices, where neighboring slices provide symmetric contextual information for identifying features on the central slice. The common strategy maps such stacks to pseudo-RGB inputs to enable transfer learning from pretrained models. However, this mapping imposes channel-specific semantics inherited from natural images, even though electron microscopy slices are homogeneous in the modality and symmetric in their predictive roles. As a result, pretrained models may encode inductive biases that are misaligned with the inherent symmetry of volumetric electron microscopy data. In this work, it is demonstrated that RGB-pretrained models systematically assign unequal importance to individual input slices when applied to stacked electron microscopy data, despite the absence of any intrinsic channel ordering. Using saliency-based attribution analysis across multiple architectures, a consistent channel-level asymmetry was observed that persists after fine-tuning and affects model interpretability, even when segmentation performance is unchanged. To address this issue, a targeted modification of pretraining weights based on uniform channel initialization was proposed, which restores symmetric feature attribution while preserving the benefits of pretraining. Experiments on the SNEMI, Lucchi and GF-PA66 datasets confirm a substantial reduction in attribution bias without compromising or even improving segmentation accuracy.