Neural Optimization Kernel: Towards Robust Deep Learning

TL;DR

This paper introduces the Neural Optimization Kernel (NOK), a novel kernel family linking deep neural networks to structured kernel methods, providing insights into generalization and robustness in over-parameterized regimes.

Contribution

It establishes a theoretical connection between deep neural networks and NOK, a new kernel family, and demonstrates its effectiveness in improving generalization and robustness.

Findings

NOK provides a new perspective on deep NN generalization.

Structured NOK blocks enhance robustness against input noise.

A new generalization bound for deep structured NOK is derived.

Abstract

Deep neural networks (NN) have achieved great success in many applications. However, why do deep neural networks obtain good generalization at an over-parameterization regime is still unclear. To better understand deep NN, we establish the connection between deep NN and a novel kernel family, i.e., Neural Optimization Kernel (NOK). The architecture of structured approximation of NOK performs monotonic descent updates of implicit regularization problems. We can implicitly choose the regularization problems by employing different activation functions, e.g., ReLU, max pooling, and soft-thresholding. We further establish a new generalization bound of our deep structured approximated NOK architecture. Our unsupervised structured approximated NOK block can serve as a simple plug-in of popular backbones for a good generalization against input noise.