Mapping DNN Embedding Manifolds for Network Generalization Prediction

TL;DR

This paper introduces a novel Network Generalization Prediction method that estimates DNN performance in new domains using only unlabeled data embeddings, achieving state-of-the-art results without domain knowledge.

Contribution

It presents the first NGP approach that predicts DNN performance solely from unlabeled image embeddings, eliminating the need for domain metadata.

Findings

Achieved state-of-the-art NGP in 13 of 15 tasks.

Effectively identified misclassified images.

Works across multiple classification domains.

Abstract

Understanding Deep Neural Network (DNN) performance in changing conditions is essential for deploying DNNs in safety critical applications with unconstrained environments, e.g., perception for self-driving vehicles or medical image analysis. Recently, the task of Network Generalization Prediction (NGP) has been proposed to predict how a DNN will generalize in a new operating domain. Previous NGP approaches have relied on labeled metadata and known distributions for the new operating domains. In this study, we propose the first NGP approach that predicts DNN performance based solely on how unlabeled images from an external operating domain map in the DNN embedding space. We demonstrate this technique for pedestrian, melanoma, and animal classification tasks and show state of the art NGP in 13 of 15 NGP tasks without requiring domain knowledge. Additionally, we show that our NGP embedding maps can be used to identify misclassified images when the DNN performance is poor.