Detection and Classification of (Pre)Cancerous Cells in Pap Smears: An Ensemble Strategy for the RIVA Cervical Cytology Challenge

TL;DR

This paper presents an ensemble detection approach for cervical cell classification in Pap smears, addressing class imbalance and nuclear overlap challenges to improve performance in the RIVA challenge.

Contribution

It introduces a systematic evaluation of three strategies—loss reweighting, data resampling, transfer learning—and combines them into an ensemble using Weighted Boxes Fusion for better detection.

Findings

Ensemble achieved a mAP50-95 of 0.201 on preliminary test.

29% improvement over best individual model on final test.

Effective mitigation of class imbalance and overlap issues.

Abstract

Automated detection and classification of cervical cells in conventional Pap smear images can strengthen cervical cancer screening at scale by reducing manual workload, improving triage, and increasing consistency across readers. However, it is challenged by severe class imbalance and frequent nuclear overlap. We present our approach to the RIVA Cervical Cytology Challenge (ISBI 2026), which requires multi-class detection of eight Bethesda cell categories under these conditions. Using YOLOv11m as the base architecture, we systematically evaluate three strategies to improve detection performance: loss reweighting, data resampling and transfer learning. We build an ensemble by combining models trained under each strategy, promoting complementary detection behavior and combining them through Weighted Boxes Fusion (WBF). The ensemble achieves a mAP50-95 of 0.201 on the preliminary test set and 0.147 on the final test set, representing a 29% improvement over the best individual model on the final test set and demonstrating the effectiveness of combining complementary imbalance mitigation strategies.