Automated Spinal MRI Labelling from Reports Using a Large Language Model

TL;DR

This paper introduces a pipeline that uses large language models to automatically extract labels from spinal MRI reports, enabling efficient annotation and training of imaging classifiers that perform comparably to those trained on manual labels.

Contribution

The study demonstrates that open-source large language models can effectively extract clinical labels from reports, matching GPT-4's performance and facilitating automated training of MRI classifiers.

Findings

Open-source models match or outperform GPT-4 on report labeling.

Automated labels enable training classifiers with comparable accuracy to manual annotations.

The pipeline is validated on five spinal conditions.

Abstract

We propose a general pipeline to automate the extraction of labels from radiology reports using large language models, which we validate on spinal MRI reports. The efficacy of our labelling method is measured on five distinct conditions: spinal cancer, stenosis, spondylolisthesis, cauda equina compression and herniation. Using open-source models, our method equals or surpasses GPT-4 on a held-out set of reports. Furthermore, we show that the extracted labels can be used to train imaging models to classify the identified conditions in the accompanying MR scans. All classifiers trained using automated labels achieve comparable performance to models trained using scans manually annotated by clinicians. Code can be found at https://github.com/robinyjpark/AutoLabelClassifier.