GNNFormer: A Graph-based Framework for Cytopathology Report Generation

TL;DR

GNNFormer is a novel graph-based framework that explicitly models cell structures in pathology images to improve automatic cytopathology report generation, outperforming existing methods.

Contribution

It introduces the first report generation method that explicitly models cell structural information using a combined GNN and Transformer framework.

Findings

Outperforms state-of-the-art baselines on NMI-WSI dataset

Effectively fuses structural, morphological, and background features

Demonstrates improved report quality and diagnostic relevance

Abstract

Cytopathology report generation is a necessary step for the standardized examination of pathology images. However, manually writing detailed reports brings heavy workloads for pathologists. To improve efficiency, some existing works have studied automatic generation of cytopathology reports, mainly by applying image caption generation frameworks with visual encoders originally proposed for natural images. A common weakness of these works is that they do not explicitly model the structural information among cells, which is a key feature of pathology images and provides significant information for making diagnoses. In this paper, we propose a novel graph-based framework called GNNFormer, which seamlessly integrates graph neural network (GNN) and Transformer into the same framework, for cytopathology report generation. To the best of our knowledge, GNNFormer is the first report generation method that explicitly models the structural information among cells in pathology images. It also effectively fuses structural information among cells, fine-grained morphology features of cells and background features to generate high-quality reports. Experimental results on the NMI-WSI dataset show that GNNFormer can outperform other state-of-the-art baselines.