HAAF: Hierarchical Adaptation and Alignment of Foundation Models for Few-Shot Pathology Anomaly Detection

TL;DR

HAAF introduces a hierarchical framework that aligns visual and semantic features at multiple levels, improving few-shot pathology anomaly detection by addressing granularity mismatches in vision-language models.

Contribution

The paper proposes a novel Cross-Level Scaled Alignment mechanism and a dual-branch inference strategy to enhance adaptation of vision-language models for fine-grained pathology detection.

Findings

Outperforms state-of-the-art methods on four benchmarks.

Effectively scales with domain-specific backbones in low-resource scenarios.

Significantly improves detection accuracy in few-shot settings.

Abstract

Precision pathology relies on detecting fine-grained morphological abnormalities within specific Regions of Interest (ROIs), as these local, texture-rich cues - rather than global slide contexts - drive expert diagnostic reasoning. While Vision-Language (V-L) models promise data efficiency by leveraging semantic priors, adapting them faces a critical Granularity Mismatch, where generic representations fail to resolve such subtle defects. Current adaptation methods often treat modalities as independent streams, failing to ground semantic prompts in ROI-specific visual contexts. To bridge this gap, we propose the Hierarchical Adaptation and Alignment Framework (HAAF). At its core is a novel Cross-Level Scaled Alignment (CLSA) mechanism that enforces a sequential calibration order: visual features first inject context into text prompts to generate content-adaptive descriptors, which then spatially guide the visual encoder to spotlight anomalies. Additionally, a dual-branch inference strategy integrates semantic scores with geometric prototypes to ensure stability in few-shot settings. Experiments on four benchmarks show HAAF significantly outperforms state-of-the-art methods and effectively scales with domain-specific backbones (e.g., CONCH) in low-resource scenarios.