Polyp Segmentation Generalisability of Pretrained Backbones

TL;DR

This study evaluates how well different pretrained backbones for polyp segmentation generalize across datasets, finding ResNet50 models often outperform ViT-B models in cross-dataset scenarios despite lower in-dataset performance.

Contribution

It extends prior work by analyzing the generalizability of pretrained backbones, highlighting that ResNet50 models tend to generalize better across datasets.

Findings

ResNet50 models show better cross-dataset generalization.

ViT-B models outperform ResNet50 on the original test set.

Pretraining pipeline effects are consistent with previous findings.

Abstract

It has recently been demonstrated that pretraining backbones in a self-supervised manner generally provides better fine-tuned polyp segmentation performance, and that models with ViT-B backbones typically perform better than models with ResNet50 backbones. In this paper, we extend this recent work to consider generalisability. I.e., we assess the performance of models on a different dataset to that used for fine-tuning, accounting for variation in network architecture and pretraining pipeline (algorithm and dataset). This reveals how well models with different pretrained backbones generalise to data of a somewhat different distribution to the training data, which will likely arise in deployment due to different cameras and demographics of patients, amongst other factors. We observe that the previous findings, regarding pretraining pipelines for polyp segmentation, hold true when considering generalisability. However, our results imply that models with ResNet50 backbones typically generalise better, despite being outperformed by models with ViT-B backbones in evaluation on the test set from the same dataset used for fine-tuning.