The BRAVO Semantic Segmentation Challenge Results in UNCV2024

TL;DR

The BRAVO challenge benchmarks semantic segmentation models' robustness and reliability under real-world perturbations and out-of-distribution scenarios, highlighting the impact of large-scale pre-training and minimal architecture.

Contribution

This paper introduces the BRAVO challenge for evaluating semantic segmentation reliability under perturbations and OOD conditions, providing new insights into model robustness.

Findings

Large-scale pre-training enhances robustness.

Minimal architectural complexity benefits reliability.

Models show varied performance in OOD detection.

Abstract

We propose the unified BRAVO challenge to benchmark the reliability of semantic segmentation models under realistic perturbations and unknown out-of-distribution (OOD) scenarios. We define two categories of reliability: (1) semantic reliability, which reflects the model's accuracy and calibration when exposed to various perturbations; and (2) OOD reliability, which measures the model's ability to detect object classes that are unknown during training. The challenge attracted nearly 100 submissions from international teams representing notable research institutions. The results reveal interesting insights into the importance of large-scale pre-training and minimal architectural design in developing robust and reliable semantic segmentation models.