Open Horizons: Evaluating Deep Models in the Wild

TL;DR

This paper conducts a comprehensive evaluation of deep models in open-world scenarios, focusing on open-set recognition and few-shot class-incremental learning, comparing various architectures and methods on CIFAR-10.

Contribution

It provides a unified experimental framework analyzing how different backbones and scoring functions impact open-world recognition and incremental learning performance.

Findings

CLIP outperforms other encoders in open-set recognition across multiple metrics.

Energy scoring function offers stable performance across different backbones.

ConCM achieves high accuracy in few-shot incremental learning, especially at 10 shots.

Abstract

Open-world deployment requires models to recognize both known categories and remain reliable when novel classes appear. We present a unified experimental study spanning open-set recognition (OSR) and few-shot class-incremental learning (FSCIL) on CIFAR-10. For OSR, we compare three pretrained frozen visual encoders: ResNet-50, ConvNeXt-Tiny and CLIP ViT-B/16,using a linear probe and four post-hoc scoring functions, namely MSP, Energy, Mahalanobis and kNN. Across metrics,such as, AUROC, AUPR, FPR@95, and OSCR, CLIP consistently yields the strongest separability between known and unknown samples, with Energy providing the most stable performance across backbones. For FSCIL, we compare modified SPPR, OrCo, and ConCM using partially frozen ResNet-50 across 1-, 5-, and 10-shot scenarios. ConCM achieves 84.7% accuracy in the 10-shot setting with the cleanest confusion matrix, while all methods show saturation beyond 5 shots. Our controlled evaluation reveals how the backbone architecture and scoring mechanisms affect unknown detection and how prototype-based methods mitigate catastrophic forgetting during incremental adaptation.