Why Deep Learning's Performance Data Are Misleading

TL;DR

This paper reveals that the impressive performance data of deep learning are often misleading due to misconducts like data deletion and test on training set, which inflate results without true generalization.

Contribution

It provides a theoretical explanation of how misconducts inflate deep learning performance metrics and introduces the NNWT classification method to illustrate this issue.

Findings

NNWT can achieve zero error using misconducts

Deep learning methods often lack true generalization

Misconducts like data deletion inflate performance metrics

Abstract

This is a theoretical paper, as a companion paper of the keynote talk at the same conference AIEE 2023. In contrast to conscious learning, many projects in AI have employed so-called "deep learning" many of which seemed to give impressive performance. This paper explains that such performance data are deceptively inflated due to two misconducts: "data deletion" and "test on training set". This paper clarifies "data deletion" and "test on training set" in deep learning and why they are misconducts. A simple classification method is defined, called Nearest Neighbor With Threshold (NNWT). A theorem is established that the NNWT method reaches a zero error on any validation set and any test set using the two misconducts, as long as the test set is in the possession of the author and both the amount of storage space and the time of training are finite but unbounded like with many deep learning methods. However, many deep learning methods, like the NNWT method, are all not generalizable since they have never been tested by a true test set. Why? The so-called "test set" was used in the Post-Selection step of the training stage. The evidence that misconducts actually took place in many deep learning projects is beyond the scope of this paper.