Novel Deep Neural Network Classifier Characterization Metrics with Applications to Dataless Evaluation

TL;DR

This paper introduces a novel method to evaluate the training quality of deep neural network classifiers without using any real dataset, relying solely on the classifier's weights and synthetic feature vectors.

Contribution

The work proposes new metrics for characterizing feature extractors and classifiers in DNNs using synthetic data and weight analysis, enabling data-less evaluation.

Findings

Data-less evaluation correlates well with traditional accuracy measures

Metrics effectively characterize feature extractor quality

Method validated on ResNet18 with CAFIR datasets

Abstract

The mainstream AI community has seen a rise in large-scale open-source classifiers, often pre-trained on vast datasets and tested on standard benchmarks; however, users facing diverse needs and limited, expensive test data may be overwhelmed by available choices. Deep Neural Network (DNN) classifiers undergo training, validation, and testing phases using example dataset, with the testing phase focused on determining the classification accuracy of test examples without delving into the inner working of the classifier. In this work, we evaluate a DNN classifier's training quality without any example dataset. It is assumed that a DNN is a composition of a feature extractor and a classifier which is the penultimate completely connected layer. The quality of a classifier is estimated using its weight vectors. The feature extractor is characterized using two metrics that utilize feature vectors it produces when synthetic data is fed as input. These synthetic input vectors are produced by backpropagating desired outputs of the classifier. Our empirical study of the proposed method for ResNet18, trained with CAFIR10 and CAFIR100 datasets, confirms that data-less evaluation of DNN classifiers is indeed possible.