Large Multimodal Model based Standardisation of Pathology Reports with Confidence and their Prognostic Significance

TL;DR

This paper introduces a large multimodal model framework that automatically extracts and standardizes information from pathology reports, providing confidence scores and demonstrating prognostic significance for patient stratification.

Contribution

It presents a novel two-stage prompting approach using LMMs for extracting and validating report data with confidence scores, applicable to diverse report formats.

Findings

Confidence scores effectively indicate extraction accuracy.

The extracted data has significant prognostic value.

Framework generalizes across multiple medical centers.

Abstract

Pathology reports are rich in clinical and pathological details but are often presented in free-text format. The unstructured nature of these reports presents a significant challenge limiting the accessibility of their content. In this work, we present a practical approach based on the use of large multimodal models (LMMs) for automatically extracting information from scanned images of pathology reports with the goal of generating a standardised report specifying the value of different fields along with estimated confidence about the accuracy of the extracted fields. The proposed approach overcomes limitations of existing methods which do not assign confidence scores to extracted fields limiting their practical use. The proposed framework uses two stages of prompting a Large Multimodal Model (LMM) for information extraction and validation. The framework generalises to textual reports from multiple medical centres as well as scanned images of legacy pathology reports. We show that the estimated confidence is an effective indicator of the accuracy of the extracted information that can be used to select only accurately extracted fields. We also show the prognostic significance of structured and unstructured data from pathology reports and show that the automatically extracted field values significant prognostic value for patient stratification. The framework is available for evaluation via the URL: https://labieb.dcs.warwick.ac.uk/.