Towards a universal mechanism for successful deep learning

TL;DR

This paper verifies a universal mechanism in deep learning where filters progressively sharpen their output clusters, increasing signal-to-noise ratio and accuracy across different architectures and datasets, suggesting potential for architecture simplification.

Contribution

It extends the understanding of a filter-based mechanism for successful deep learning across multiple architectures and datasets, demonstrating its universality and implications for model simplification.

Findings

Accuracy and SNR increase with depth in models.

Error rate scales linearly with number of labels.

Mechanism is consistent across datasets from 3 to 1000 labels.

Abstract

Recently, the underlying mechanism for successful deep learning (DL) was presented based on a quantitative method that measures the quality of a single filter in each layer of a DL model, particularly VGG-16 trained on CIFAR-10. This method exemplifies that each filter identifies small clusters of possible output labels, with additional noise selected as labels outside the clusters. This feature is progressively sharpened with each layer, resulting in an enhanced signal-to-noise ratio (SNR), which leads to an increase in the accuracy of the DL network. In this study, this mechanism is verified for VGG-16 and EfficientNet-B0 trained on the CIFAR-100 and ImageNet datasets, and the main results are as follows. First, the accuracy and SNR progressively increase with the layers. Second, for a given deep architecture, the maximal error rate increases approximately linearly with the number of output labels. Third, similar trends were obtained for dataset labels in the range [3, 1,000], thus supporting the universality of this mechanism. Understanding the performance of a single filter and its dominating features paves the way to highly dilute the deep architecture without affecting its overall accuracy, and this can be achieved by applying the filter's cluster connections (AFCC).