Subject2Vec: Generative-Discriminative Approach from a Set of Image Patches to a Vector

TL;DR

Subject2Vec introduces an attention-based, set-to-vector method for disease severity prediction from variable-length image patches, providing interpretability and state-of-the-art accuracy in COPD lung CT analysis.

Contribution

It presents a novel generative-discriminative model that handles variable input sizes and offers interpretability through attention mechanisms in medical imaging.

Findings

Achieves state-of-the-art COPD severity prediction accuracy.

Provides interpretable attention maps highlighting relevant lung regions.

Handles variable-length image inputs without resizing.

Abstract

We propose an attention-based method that aggregates local image features to a subject-level representation for predicting disease severity. In contrast to classical deep learning that requires a fixed dimensional input, our method operates on a set of image patches; hence it can accommodate variable length input image without image resizing. The model learns a clinically interpretable subject-level representation that is reflective of the disease severity. Our model consists of three mutually dependent modules which regulate each other: (1) a discriminative network that learns a fixed-length representation from local features and maps them to disease severity; (2) an attention mechanism that provides interpretability by focusing on the areas of the anatomy that contribute the most to the prediction task; and (3) a generative network that encourages the diversity of the local latent features. The generative term ensures that the attention weights are non-degenerate while maintaining the relevance of the local regions to the disease severity. We train our model end-to-end in the context of a large-scale lung CT study of Chronic Obstructive Pulmonary Disease (COPD). Our model gives state-of-the art performance in predicting clinical measures of severity for COPD. The distribution of the attention provides the regional relevance of lung tissue to the clinical measurements.