DALPHIN: Benchmarking Digital Pathology AI Copilots Against Pathologists on an Open Multicentric Dataset

TL;DR

DALPHIN introduces a comprehensive open benchmark dataset to evaluate AI copilots in digital pathology, comparing their performance against expert pathologists across diverse cases and diagnoses.

Contribution

It provides the first multicentric open benchmark dataset for pathology AI copilots, including performance evaluation against human experts.

Findings

PathChat+ performed comparably to experts in 4 out of 6 tasks.

GPT-5 showed no significant difference from experts in 1 out of 6 tasks.

DALPHIN dataset covers 130 diagnoses, 6 countries, and 14 subspecialties.

Abstract

Foundation models with visual question answering capabilities for digital pathology are emerging. Such unprecedented technology requires independent benchmarking to assess its potential in assisting pathologists in routine diagnostics. We created DALPHIN, the first multicentric open benchmark for pathology AI copilots, comprising 1236 images from 300 cases, spanning 130 rare to common diagnoses, 6 countries, and 14 subspecialties. The DALPHIN design and dataset are introduced alongside a human performance benchmark of 31 pathologists from 10 countries with varying expertise. We report results for two general-purpose (GPT-5, Gemini 2.5 Pro) and one pathology-specific copilot (PathChat+) for sequential and independent answer generation. We observed no statistically significant difference from expert-level performance in four of six tasks for PathChat, 2/6 tasks for Gemini, and 1/6 tasks for GPT. DALPHIN is publicly released with sequestered, indirectly accessible ground truth to foster robust and enduring benchmarking. Data, methods, and the evaluation platform are accessible through dalphin.grand-challenge.org.