Interleaved Text/Image Deep Mining on a Large-Scale Radiology Database for Automated Image Interpretation

TL;DR

This paper introduces a large-scale deep learning system that combines text and image analysis to automatically interpret radiology images by extracting semantic information and disease predictions from a vast medical database.

Contribution

It presents a novel interleaved text/image deep learning approach for mining and interpreting large-scale radiology data, integrating unsupervised and supervised learning for semantic labeling.

Findings

Successfully matched 216K images with reports

Predicted semantic topics and keywords for radiology images

Detected common disease types with high accuracy

Abstract

Despite tremendous progress in computer vision, there has not been an attempt for machine learning on very large-scale medical image databases. We present an interleaved text/image deep learning system to extract and mine the semantic interactions of radiology images and reports from a national research hospital's Picture Archiving and Communication System. With natural language processing, we mine a collection of representative ~216K two-dimensional key images selected by clinicians for diagnostic reference, and match the images with their descriptions in an automated manner. Our system interleaves between unsupervised learning and supervised learning on document- and sentence-level text collections, to generate semantic labels and to predict them given an image. Given an image of a patient scan, semantic topics in radiology levels are predicted, and associated key-words are generated. Also, a number of frequent disease types are detected as present or absent, to provide more specific interpretation of a patient scan. This shows the potential of large-scale learning and prediction in electronic patient records available in most modern clinical institutions.