Pathology Report Generation and Multimodal Representation Learning for Cutaneous Melanocytic Lesions

TL;DR

This paper introduces a vision-language model tailored for pathology report generation of skin lesions, demonstrating comparable quality to expert reports for common cases and improved retrieval for rare subtypes.

Contribution

It presents a novel contrastive captioning model trained on a large dataset of skin lesion images and reports, advancing automated pathology report generation.

Findings

Model reports are comparable to pathologists for common nevi.

Cross-modal retrieval outperforms in rare lesion cases.

Report generation is more challenging for rare subtypes.

Abstract

Millions of melanocytic skin lesions are examined by pathologists each year, the majority of which concern common nevi (i.e., ordinary moles). While most of these lesions can be diagnosed in seconds, writing the corresponding pathology report is much more time-consuming. Automating part of the report writing could, therefore, alleviate the increasing workload of pathologists. In this work, we develop a vision-language model specifically for the pathology domain of cutaneous melanocytic lesions. The model follows the Contrastive Captioner framework and was trained and evaluated using a melanocytic lesion dataset of 42,512 H&E-stained whole slide images and 19,645 corresponding pathology reports. Our results show that the quality scores of model-generated reports were on par with pathologist-written reports for common nevi, assessed by an expert pathologist in a reader study. While report generation revealed to be more difficult for rare melanocytic lesion subtypes, the cross-modal retrieval performance for these cases was considerably better.