Improved statistical benchmarking of digital pathology models using pairwise frames evaluation

TL;DR

This paper introduces a nested pairwise frames evaluation method for benchmarking digital pathology models, addressing data size and variability issues by comparing model agreement with pathologists and among pathologists.

Contribution

The paper presents a novel nested pairwise frames evaluation framework for more reliable benchmarking of digital pathology models against manual annotations.

Findings

Effective in tissue and cell classification tasks

Addresses data size limitations in model validation

Provides consistent agreement metrics across models

Abstract

Nested pairwise frames is a method for relative benchmarking of cell or tissue digital pathology models against manual pathologist annotations on a set of sampled patches. At a high level, the method compares agreement between a candidate model and pathologist annotations with agreement among pathologists' annotations. This evaluation framework addresses fundamental issues of data size and annotator variability in using manual pathologist annotations as a source of ground truth for model validation. We implemented nested pairwise frames evaluation for tissue classification, cell classification, and cell count prediction tasks and show results for cell and tissue models deployed on an H&E-stained melanoma dataset.