Learning Class Difficulty in Imbalanced Histopathology Segmentation via Dynamic Focal Attention

TL;DR

This paper introduces Dynamic Focal Attention, a novel attention-based mechanism that learns class-specific difficulty in histopathology segmentation, improving performance without extra training stages.

Contribution

It proposes a learnable class-specific bias within cross-attention to adaptively encode class difficulty, unifying frequency and difficulty-aware approaches.

Findings

DFA improves Dice and IoU on three histopathology benchmarks.

DFA matches or exceeds difficulty-aware baselines without additional training.

Representation-level difficulty encoding is a principled alternative to loss reweighting.

Abstract

Semantic segmentation of histopathology images under class imbalance is typically addressed through frequency-based loss reweighting, which implicitly assumes that rare classes are difficult. However, true difficulty also arises from morphological variability, boundary ambiguity, and contextual similarity-factors that frequency cannot capture. We propose Dynamic Focal Attention (DFA), a simple and efficient mechanism that learns class-specific difficulty directly within the cross-attention of query-based mask decoders. DFA introduces a learnable per-class bias to attention logits, enabling representation-level reweighting prior to prediction rather than gradient-level reweighting after prediction. Initialised from a log-frequency prior to prevent gradient starvation, the bias is optimised end-to-end, allowing the model to adaptively capture difficulty signals through training, effectively unifying frequency-based and difficulty-aware approaches under a common attention-bias framework. On three histopathology benchmarks (BDSA, BCSS, CRAG), DFA consistently improves Dice and IoU, matching or exceeding a difficulty-aware baseline without a separate estimator or additional training stage. These results demonstrate that encoding class difficulty at the representation level provides a principled alternative to conventional loss reweighting for imbalanced segmentation.