On the Importance of Text Preprocessing for Multimodal Representation Learning and Pathology Report Generation

TL;DR

This study examines how preprocessing pathology reports to include only relevant image-derived information improves the accuracy and reduces hallucinations in multimodal pathology report generation, highlighting the importance of text selection.

Contribution

It demonstrates that text preprocessing enhances report quality and reduces hallucinations, providing insights into optimal data preparation for vision-language models in pathology.

Findings

Preprocessed reports prevent hallucinated sentences in generated reports.

Training on full reports improves cross-modal retrieval performance.

Preprocessing enhances report quality despite lower retrieval accuracy.

Abstract

Vision-language models in pathology enable multimodal case retrieval and automated report generation. Many of the models developed so far, however, have been trained on pathology reports that include information which cannot be inferred from paired whole slide images (e.g., patient history), potentially leading to hallucinated sentences in generated reports. To this end, we investigate how the selection of information from pathology reports for vision-language modeling affects the quality of the multimodal representations and generated reports. More concretely, we compare a model trained on full reports against a model trained on preprocessed reports that only include sentences describing the cell and tissue appearances based on the H&E-stained slides. For the experiments, we built upon the BLIP-2 framework and used a cutaneous melanocytic lesion dataset of 42,433 H&E-stained whole slide images and 19,636 corresponding pathology reports. Model performance was assessed using image-to-text and text-to-image retrieval, as well as qualitative evaluation of the generated reports by an expert pathologist. Our results demonstrate that text preprocessing prevents hallucination in report generation. Despite the improvement in the quality of the generated reports, training the vision-language model on full reports showed better cross-modal retrieval performance.