Label Convergence: Defining an Upper Performance Bound in Object Recognition through Contradictory Annotations

TL;DR

This paper introduces the concept of label convergence to define an upper bound on object recognition performance considering annotation contradictions, highlighting the impact of label noise on model evaluation and suggesting directions for future data quality improvements.

Contribution

It formalizes the notion of label convergence as an upper bound on model accuracy in the presence of contradictory annotations, supported by analysis of real-world datasets including LVIS.

Findings

Label convergence for LVIS is approximately 62.63-67.52 mAP.

State-of-the-art models reach the upper end of the label convergence interval.

Annotation errors significantly influence achievable performance bounds.

Abstract

Annotation errors are a challenge not only during training of machine learning models, but also during their evaluation. Label variations and inaccuracies in datasets often manifest as contradictory examples that deviate from established labeling conventions. Such inconsistencies, when significant, prevent models from achieving optimal performance on metrics such as mean Average Precision (mAP). We introduce the notion of "label convergence" to describe the highest achievable performance under the constraint of contradictory test annotations, essentially defining an upper bound on model accuracy. Recognizing that noise is an inherent characteristic of all data, our study analyzes five real-world datasets, including the LVIS dataset, to investigate the phenomenon of label convergence. We approximate that label convergence is between 62.63-67.52 mAP@[0.5:0.95:0.05] for LVIS with 95% confidence, attributing these bounds to the presence of real annotation errors. With current state-of-the-art (SOTA) models at the upper end of the label convergence interval for the well-studied LVIS dataset, we conclude that model capacity is sufficient to solve current object detection problems. Therefore, future efforts should focus on three key aspects: (1) updating the problem specification and adjusting evaluation practices to account for unavoidable label noise, (2) creating cleaner data, especially test data, and (3) including multi-annotated data to investigate annotation variation and make these issues visible from the outset.