On "Deep Learning" Misconduct

TL;DR

This theoretical paper highlights a misconduct in deep learning where models are tested on training data, violating protocols and undermining generalization, demonstrated through a theorem involving the PGNN method.

Contribution

The paper establishes a theorem showing that common misconduct allows perfect training and validation errors, challenging the generalizability of deep learning models.

Findings

PGNN can achieve zero error on validation and test sets.

Misconduct violates transparency and cross-validation protocols.

Deep learning models may not be genuinely generalizable.

Abstract

This is a theoretical paper, as a companion paper of the plenary talk for the same conference ISAIC 2022. In contrast to the author's plenary talk in the same conference, conscious learning (Weng, 2022b; Weng, 2022c) which develops a single network for a life (many tasks), "Deep Learning" trains multiple networks for each task. Although "Deep Learning" may use different learning modes, including supervised, reinforcement and adversarial modes, almost all "Deep Learning" projects apparently suffer from the same misconduct, called "data deletion" and "test on training data". This paper establishes a theorem that a simple method called Pure-Guess Nearest Neighbor (PGNN) reaches any required errors on validation data set and test data set, including zero-error requirements, through the same misconduct, as long as the test data set is in the possession of the authors and both the amount of storage space and the time of training are finite but unbounded. The misconduct violates well-known protocols called transparency and cross-validation. The nature of the misconduct is fatal, because in the absence of any disjoint test, "Deep Learning" is clearly not generalizable.