HiPath: Hierarchical Vision-Language Alignment for Structured Pathology Report Prediction

TL;DR

HiPath is a hierarchical vision-language model designed for structured pathology report prediction, effectively encoding multi-granular diagnostic information from pathology images and reports, outperforming baselines on real-world Chinese datasets.

Contribution

The paper introduces HiPath, a novel lightweight framework with hierarchical modules for structured pathology report prediction using frozen vision-language backbones.

Findings

Achieves 68.9% strict accuracy on real-world data.

Maintains high safety rate of 97.3%.

Generalizes well across hospitals with minimal accuracy drop.

Abstract

Pathology reports are structured, multi-granular documents encoding diagnostic conclusions, histological grades, and ancillary test results across one or more anatomical sites; yet existing pathology vision-language models (VLMs) reduce this output to a flat label or free-form text. We present HiPath, a lightweight VLM framework built on frozen UNI2 and Qwen3 backbones that treats structured report prediction as its primary training objective. Three trainable modules totalling 15M parameters address complementary aspects of the problem: a Hierarchical Patch Aggregator (HiPA) for multi-image visual encoding, Hierarchical Contrastive Learning (HiCL) for cross-modal alignment via optimal transport, and Slot-based Masked Diagnosis Prediction (Slot-MDP) for structured diagnosis generation. Trained on 749K real-world Chinese pathology cases from three hospitals, HiPath achieves 68.9% strict and 74.7% clinically acceptable accuracy with a 97.3% safety rate, outperforming all baselines under the same frozen backbone. Cross-hospital evaluation confirms generalisation with only a 3.4pp drop in strict accuracy while maintaining 97.1% safety.