Nodule2vec: a 3D Deep Learning System for Pulmonary Nodule Retrieval Using Semantic Representation

TL;DR

This paper introduces Nodule2vec, a deep learning system that converts 3D pulmonary nodule images into semantic vectors for content-based retrieval, aiding radiologists by providing similar case references.

Contribution

It presents a novel 3D deep learning approach for pulmonary nodule retrieval that preserves semantic information and overcomes dataset limitations through transfer learning.

Findings

The system accurately measures similarity between nodules.

It performs comparably to a second radiologist opinion.

The approach enhances decision support for radiologists.

Abstract

Content-based retrieval supports a radiologist decision making process by presenting the doctor the most similar cases from the database containing both historical diagnosis and further disease development history. We present a deep learning system that transforms a 3D image of a pulmonary nodule from a CT scan into a low-dimensional embedding vector. We demonstrate that such a vector representation preserves semantic information about the nodule and offers a viable approach for content-based image retrieval (CBIR). We discuss the theoretical limitations of the available datasets and overcome them by applying transfer learning of the state-of-the-art lung nodule detection model. We evaluate the system using the LIDC-IDRI dataset of thoracic CT scans. We devise a similarity score and show that it can be utilized to measure similarity 1) between annotations of the same nodule by different radiologists and 2) between the query nodule and the top four CBIR results. A comparison between doctors and algorithm scores suggests that the benefit provided by the system to the radiologist end-user is comparable to obtaining a second radiologist's opinion.