Fine-grained lesion annotation in CT images with knowledge mined from radiology reports

TL;DR

This paper presents a method for detailed lesion annotation in CT images by mining labels from radiology reports, using a multi-label CNN to predict lesion attributes with high accuracy, aiding computer-aided diagnosis.

Contribution

It introduces a novel approach to automatically mine lesion labels from reports and applies a multi-label CNN for fine-grained lesion annotation in CT images.

Findings

Achieved an average ROC AUC of 0.9083 on test lesions.

Demonstrated the effectiveness of label mining from reports.

Validated the multi-label CNN approach for lesion annotation.

Abstract

In radiologists' routine work, one major task is to read a medical image, e.g., a CT scan, find significant lesions, and write sentences in the radiology report to describe them. In this paper, we study the lesion description or annotation problem as an important step of computer-aided diagnosis (CAD). Given a lesion image, our aim is to predict multiple relevant labels, such as the lesion's body part, type, and attributes. To address this problem, we define a set of 145 labels based on RadLex to describe a large variety of lesions in the DeepLesion dataset. We directly mine training labels from the lesion's corresponding sentence in the radiology report, which requires minimal manual effort and is easily generalizable to large data and label sets. A multi-label convolutional neural network is then proposed for images with multi-scale structure and a noise-robust loss. Quantitative and qualitative experiments demonstrate the effectiveness of the framework. The average area under ROC curve on 1,872 test lesions is 0.9083.