Implicit Hypothesis Testing and Divergence Preservation in Neural Network Representations

TL;DR

This paper analyzes neural network training dynamics using hypothesis testing, revealing that well-generalizing models approach optimal decision rules by preserving divergence in learned representations.

Contribution

It introduces a hypothesis testing perspective to neural training, demonstrating divergence preservation as a marker of optimality and proposing an informational plane for convergence assessment.

Findings

Networks approach Neyman-Pearson optimal decision rules during training

Monotonic growth in KL divergence correlates with better generalization

Defines an Evidence-Error plane to evaluate convergence across architectures

Abstract

We study the training dynamics of neural classifiers through the lens of binary hypothesis testing. We re-formalize classification as a collection of binary tests between class-conditional distributions induced by learned representations and show empirically that, along training trajectories, well-generalizing networks progressively approach Neyman-Pearson optimal decision rules, as measured by monotonic growth in the KL divergence retained by learned representations. We provide sufficient conditions for exact optimality, discuss its implications for training regularization, and define an informational plane, (so-called Evidence-Error plane) where convergence can be assessed methodically across network architecture.