WSI-VQA: Interpreting Whole Slide Images by Generative Visual Question Answering

TL;DR

This paper introduces WSI-VQA, a generative visual question answering framework for interpreting whole slide images, enabling diverse diagnostic tasks and outperforming existing models with a new dataset and explainability features.

Contribution

The paper presents a novel generative VQA framework for WSIs, a new dataset with question-answer pairs, and demonstrates improved performance and interpretability over existing methods.

Findings

W2T model outperforms discriminative models in medical correctness.

A new dataset with 8672 question-answer pairs for WSIs is established.

Visual co-attention provides intuitive explanations for diagnostic results.

Abstract

Whole slide imaging is routinely adopted for carcinoma diagnosis and prognosis. Abundant experience is required for pathologists to achieve accurate and reliable diagnostic results of whole slide images (WSI). The huge size and heterogeneous features of WSIs make the workflow of pathological reading extremely time-consuming. In this paper, we propose a novel framework (WSI-VQA) to interpret WSIs by generative visual question answering. WSI-VQA shows universality by reframing various kinds of slide-level tasks in a question-answering pattern, in which pathologists can achieve immunohistochemical grading, survival prediction, and tumor subtyping following human-machine interaction. Furthermore, we establish a WSI-VQA dataset which contains 8672 slide-level question-answering pairs with 977 WSIs. Besides the ability to deal with different slide-level tasks, our generative model which is named Wsi2Text Transformer (W2T) outperforms existing discriminative models in medical correctness, which reveals the potential of our model to be applied in the clinical scenario. Additionally, we also visualize the co-attention mapping between word embeddings and WSIs as an intuitive explanation for diagnostic results. The dataset and related code are available at https://github.com/cpystan/WSI-VQA.