SemSegBench & DetecBench: Benchmarking Reliability and Generalization Beyond Classification

TL;DR

This paper introduces benchmarking tools SEMSEGBENCH and DETECBENCH to evaluate the robustness and generalization of segmentation and detection models under adversarial attacks and corruptions, revealing systematic weaknesses and guiding future improvements.

Contribution

It provides the most extensive evaluation to date of segmentation and detection models' reliability, along with open-source benchmarking tools and a large dataset of evaluations for future research.

Findings

Systematic weaknesses in state-of-the-art models under adversarial attacks.

Key trends related to architecture, backbone, and model capacity.

Benchmarking results across multiple datasets and models.

Abstract

Reliability and generalization in deep learning are predominantly studied in the context of image classification. Yet, real-world applications in safety-critical domains involve a broader set of semantic tasks, such as semantic segmentation and object detection, which come with a diverse set of dedicated model architectures. To facilitate research towards robust model design in segmentation and detection, our primary objective is to provide benchmarking tools regarding robustness to distribution shifts and adversarial manipulations. We propose the benchmarking tools SEMSEGBENCH and DETECBENCH, along with the most extensive evaluation to date on the reliability and generalization of semantic segmentation and object detection models. In particular, we benchmark 76 segmentation models across four datasets and 61 object detectors across two datasets, evaluating their performance under diverse adversarial attacks and common corruptions. Our findings reveal systematic weaknesses in state-of-the-art models and uncover key trends based on architecture, backbone, and model capacity. SEMSEGBENCH and DETECBENCH are open-sourced in our GitHub repository (https://github.com/shashankskagnihotri/benchmarking_reliability_generalization) along with our complete set of total 6139 evaluations. We anticipate the collected data to foster and encourage future research towards improved model reliability beyond classification.