Multiple Instance Captioning: Learning Representations from Histopathology Textbooks and Articles

TL;DR

This paper introduces ARCH, a comprehensive pathology captioning dataset with dense descriptions, demonstrating that models trained on it transfer well to various pathology tasks, surpassing traditional ImageNet features.

Contribution

The paper presents ARCH, a novel dense-captioning dataset for computational pathology, and shows its representations transfer effectively across multiple pathology tasks.

Findings

ARCH rivals MS-COCO in intrinsic dimensionality

Pre-trained ARCH models outperform ImageNet features in pathology tasks

ARCH-based representations transfer better than self-supervised or multi-task learned features

Abstract

We present ARCH, a computational pathology (CP) multiple instance captioning dataset to facilitate dense supervision of CP tasks. Existing CP datasets focus on narrow tasks; ARCH on the other hand contains dense diagnostic and morphological descriptions for a range of stains, tissue types and pathologies. Using intrinsic dimensionality estimation, we show that ARCH is the only CP dataset to (ARCH-)rival its computer vision analog MS-COCO Captions. We conjecture that an encoder pre-trained on dense image captions learns transferable representations for most CP tasks. We support the conjecture with evidence that ARCH representation transfers to a variety of pathology sub-tasks better than ImageNet features or representations obtained via self-supervised or multi-task learning on pathology images alone. We release our best model and invite other researchers to test it on their CP tasks.