Multi-Attribute Open Set Recognition

TL;DR

This paper extends open set recognition to a multi-attribute scenario, enabling simultaneous classification and explanation of unknown samples based on specific visual attributes, highlighting challenges with spurious correlations.

Contribution

It introduces a new multi-attribute OSR problem setup and analyzes baseline vulnerabilities to spurious correlations affecting OOD detection.

Findings

Baseline methods are vulnerable to shortcuts due to spurious correlations.

Cross-attribute correlations impair OOD detection performance.

Empirical evidence shows consistent behavior across datasets.

Abstract

Open Set Recognition (OSR) extends image classification to an open-world setting, by simultaneously classifying known classes and identifying unknown ones. While conventional OSR approaches can detect Out-of-Distribution (OOD) samples, they cannot provide explanations indicating which underlying visual attribute(s) (e.g., shape, color or background) cause a specific sample to be unknown. In this work, we introduce a novel problem setup that generalizes conventional OSR to a multi-attribute setting, where multiple visual attributes are simultaneously recognized. Here, OOD samples can be not only identified but also categorized by their unknown attribute(s). We propose simple extensions of common OSR baselines to handle this novel scenario. We show that these baselines are vulnerable to shortcuts when spurious correlations exist in the training dataset. This leads to poor OOD performance which, according to our experiments, is mainly due to unintended cross-attribute correlations of the predicted confidence scores. We provide an empirical evidence showing that this behavior is consistent across different baselines on both synthetic and real world datasets.