Confidence Dimension for Deep Learning based on Hoeffding Inequality and Relative Evaluation

TL;DR

This paper introduces the confidence dimension (CD), a new measure based on Hoeffding's inequality and VC-dimension, to evaluate and rank the relative generalization ability of deep neural networks, including binary neural networks.

Contribution

It proposes a novel confidence dimension (CD) framework that theoretically estimates the upper bound of generalization for DNNs and BNNs, providing a reliable ranking method.

Findings

CD reflects relative generalization ability across DNNs and BNNs.

Experimental results validate CD's effectiveness on image classification and object detection.

CD offers a consistent measure for both full-precision and binary neural networks.

Abstract

Research on the generalization ability of deep neural networks (DNNs) has recently attracted a great deal of attention. However, due to their complex architectures and large numbers of parameters, measuring the generalization ability of specific DNN models remains an open challenge. In this paper, we propose to use multiple factors to measure and rank the relative generalization of DNNs based on a new concept of confidence dimension (CD). Furthermore, we provide a feasible framework in our CD to theoretically calculate the upper bound of generalization based on the conventional Vapnik-Chervonenk dimension (VC-dimension) and Hoeffding's inequality. Experimental results on image classification and object detection demonstrate that our CD can reflect the relative generalization ability for different DNNs. In addition to full-precision DNNs, we also analyze the generalization ability of binary neural networks (BNNs), whose generalization ability remains an unsolved problem. Our CD yields a consistent and reliable measure and ranking for both full-precision DNNs and BNNs on all the tasks.