NCT-CRC-HE: Not All Histopathological Datasets Are Equally Useful

TL;DR

This paper critically examines the NCT-CRC-HE dataset for colorectal cancer histopathology, revealing biases and artifacts that inflate model performance, and demonstrates that simple models can achieve high accuracy, questioning the dataset's reliability.

Contribution

The study identifies dataset biases and artifacts affecting histopathological image analysis and shows that simple models outperform complex ones on this dataset, highlighting issues in current evaluation practices.

Findings

Dataset contains color normalization issues and JPEG artifacts.

Simple models achieve over 50% accuracy, complex models over 97.7%.

The dataset's biases may overestimate model performance.

Abstract

Numerous deep learning-based solutions have been proposed for histopathological image analysis over the past years. While they usually demonstrate exceptionally high accuracy, one key question is whether their precision might be affected by low-level image properties not related to histopathology but caused by microscopy image handling and pre-processing. In this paper, we analyze a popular NCT-CRC-HE-100K colorectal cancer dataset used in numerous prior works and show that both this dataset and the obtained results may be affected by data-specific biases. The most prominent revealed dataset issues are inappropriate color normalization, severe JPEG artifacts inconsistent between different classes, and completely corrupted tissue samples resulting from incorrect image dynamic range handling. We show that even the simplest model using only 3 features per image (red, green and blue color intensities) can demonstrate over 50% accuracy on this 9-class dataset, while using color histogram not explicitly capturing cell morphology features yields over 82% accuracy. Moreover, we show that a basic EfficientNet-B0 ImageNet pretrained model can achieve over 97.7% accuracy on this dataset, outperforming all previously proposed solutions developed for this task, including dedicated foundation histopathological models and large cell morphology-aware neural networks. The NCT-CRC-HE dataset is publicly available and can be freely used to replicate the presented results. The codes and pre-trained models used in this paper are available at https://github.com/gmalivenko/NCT-CRC-HE-experiments