Multimodal Generalized Zero Shot Learning for Gleason Grading using Self-Supervised Learning

TL;DR

This paper introduces a novel multimodal zero-shot learning approach using self-supervised learning and generative models to predict prostate cancer Gleason grades from MRI images, bypassing the need for extensive labeled data.

Contribution

It presents a new method combining CVAE and cycle GANs for generating features of unseen grades, enabling accurate Gleason grading from MRI in a zero-shot setting.

Findings

Outperforms existing feature generation methods for GZSL.

Achieves near fully supervised performance in Gleason grading.

Demonstrates effectiveness of multimodal feature synthesis for medical diagnosis.

Abstract

Gleason grading from histopathology images is essential for accurate prostate cancer (PCa) diagnosis. Since such images are obtained after invasive tissue resection quick diagnosis is challenging under the existing paradigm. We propose a method to predict Gleason grades from magnetic resonance (MR) images which are non-interventional and easily acquired. We solve the problem in a generalized zero-shot learning (GZSL) setting since we may not access training images of every disease grade. Synthetic MRI feature vectors of unseen grades (classes) are generated by exploiting Gleason grades' ordered nature through a conditional variational autoencoder (CVAE) incorporating self-supervised learning. Corresponding histopathology features are generated using cycle GANs, and combined with MR features to predict Gleason grades of test images. Experimental results show our method outperforms competing feature generating approaches for GZSL, and comes close to performance of fully supervised methods.