Leveraging Multi-Rater Annotations to Calibrate Object Detectors in Microscopy Imaging

TL;DR

This paper presents a novel method to improve the calibration of microscopy object detectors by leveraging multi-rater annotations, leading to more reliable biomedical imaging models.

Contribution

It introduces a rater-specific ensemble approach that models inter-rater variability to enhance calibration without sacrificing detection accuracy.

Findings

Improved calibration performance with multi-rater based models

Maintained detection accuracy comparable to traditional methods

Explicit modeling of rater disagreement enhances trustworthiness

Abstract

Deep learning-based object detectors have achieved impressive performance in microscopy imaging, yet their confidence estimates often lack calibration, limiting their reliability for biomedical applications. In this work, we introduce a new approach to improve model calibration by leveraging multi-rater annotations. We propose to train separate models on the annotations from single experts and aggregate their predictions to emulate consensus. This improves upon label sampling strategies, where models are trained on mixed annotations, and offers a more principled way to capture inter-rater variability. Experiments on a colorectal organoid dataset annotated by two experts demonstrate that our rater-specific ensemble strategy improves calibration performance while maintaining comparable detection accuracy. These findings suggest that explicitly modelling rater disagreement can lead to more trustworthy object detectors in biomedical imaging.