Bimodal network architectures for automatic generation of image annotation from text

TL;DR

This paper introduces two deep neural network architectures that automatically generate image annotations from text reports, significantly reducing the need for costly manual labeling in medical imaging.

Contribution

It proposes two novel bimodal neural network architectures for automatic ROI annotation in medical images using text, with one leveraging pre-trained image features for improved accuracy.

Findings

Both architectures successfully estimate ROIs from text in chest X-ray images.

The pre-trained image network architecture outperforms the other, with ROI centroids within 5.1% of ground truth.

The methods are validated against clinical annotations, demonstrating practical applicability.

Abstract

Medical image analysis practitioners have embraced big data methodologies. This has created a need for large annotated datasets. The source of big data is typically large image collections and clinical reports recorded for these images. In many cases, however, building algorithms aimed at segmentation and detection of disease requires a training dataset with markings of the areas of interest on the image that match with the described anomalies. This process of annotation is expensive and needs the involvement of clinicians. In this work we propose two separate deep neural network architectures for automatic marking of a region of interest (ROI) on the image best representing a finding location, given a textual report or a set of keywords. One architecture consists of LSTM and CNN components and is trained end to end with images, matching text, and markings of ROIs for those images. The output layer estimates the coordinates of the vertices of a polygonal region. The second architecture uses a network pre-trained on a large dataset of the same image types for learning feature representations of the findings of interest. We show that for a variety of findings from chest X-ray images, both proposed architectures learn to estimate the ROI, as validated by clinical annotations. There is a clear advantage obtained from the architecture with pre-trained imaging network. The centroids of the ROIs marked by this network were on average at a distance equivalent to 5.1% of the image width from the centroids of the ground truth ROIs.