Variational Kernel Design for Internal Noise: Gaussian Chaos Noise, Representation Compatibility, and Reliable Deep Learning

TL;DR

This paper introduces Variational Kernel Design (VKD), a framework for internal noise in deep networks that optimizes noise correlation geometry to improve model calibration and robustness, especially using Gaussian Chaos Noise (GCh).

Contribution

It proposes a novel VKD framework that designs internal noise with optimal correlation geometry, deriving Gaussian Chaos Noise with improved calibration and robustness in deep learning models.

Findings

GCh improves calibration on ImageNet and ImageNet-C.

VKD optimizes noise correlation for better representation compatibility.

GCh enhances model robustness and accuracy under distribution shifts.

Abstract

Internal noise in deep networks is usually inherited from heuristics such as dropout, hard masking, or additive perturbation. We ask two questions: what correlation geometry should internal noise have, and is the implemented perturbation compatible with the representations it acts on? We answer these questions through Variational Kernel Design (VKD), a framework in which a noise mechanism is specified by a law family, a correlation kernel, and an injection operator, and is derived from learning desiderata. In a solved spatial subfamily, a quadratic maximum-entropy principle over latent log-fields yields a Gaussian optimizer with precision given by the Dirichlet Laplacian, so the induced geometry is the Dirichlet Green kernel. Wick normalization then gives a canonical positive mean-one gate, Gaussian Chaos Noise (GCh). For the sample-wise gate used in practice, we prove exact Gaussian control of pairwise log-ratio deformation, margin-sensitive ranking stability, and an exact expected intrinsic roughness budget; hard binary masks instead induce singular or coherence-amplified distortions on positive coherent representations. On ImageNet and ImageNet-C, GCh consistently improves calibration and under shift also improves NLL at competitive accuracy.