A Data-Driven Guided Decoding Mechanism for Diagnostic Captioning

TL;DR

This paper introduces a data-driven guided decoding approach that integrates medical tags into diagnostic captioning models to improve the accuracy of generated medical reports from images.

Contribution

It presents a novel guided decoding mechanism that incorporates medical condition tags into the caption generation process, enhancing diagnostic text accuracy.

Findings

Improved performance across multiple datasets and models.

Effective in few- and zero-shot learning scenarios.

Open-source implementation available.

Abstract

Diagnostic Captioning (DC) automatically generates a diagnostic text from one or more medical images (e.g., X-rays, MRIs) of a patient. Treated as a draft, the generated text may assist clinicians, by providing an initial estimation of the patient's condition, speeding up and helping safeguard the diagnostic process. The accuracy of a diagnostic text, however, strongly depends on how well the key medical conditions depicted in the images are expressed. We propose a new data-driven guided decoding method that incorporates medical information, in the form of existing tags capturing key conditions of the image(s), into the beam search of the diagnostic text generation process. We evaluate the proposed method on two medical datasets using four DC systems that range from generic image-to-text systems with CNN encoders and RNN decoders to pre-trained Large Language Models. The latter can also be used in few- and zero-shot learning scenarios. In most cases, the proposed mechanism improves performance with respect to all evaluation measures. We provide an open-source implementation of the proposed method at https://github.com/nlpaueb/dmmcs.