Takeuchi's Information Criteria as Generalization Measures for DNNs Close to NTK Regime

TL;DR

This paper investigates the effectiveness of Takeuchi's information criterion (TIC) as a measure of generalization in deep neural networks, especially near the neural tangent kernel (NTK) regime, through theoretical analysis and extensive experiments.

Contribution

It establishes theoretical conditions under which TIC explains DNN generalization and empirically validates TIC's correlation with generalization gaps near the NTK regime.

Findings

TIC correlates well with generalization gaps near NTK regime

Outside NTK regime, TIC's correlation with generalization gap diminishes

TIC improves trial pruning for hyperparameter optimization

Abstract

Generalization measures have been studied extensively in the machine learning community to better characterize generalization gaps. However, establishing a reliable generalization measure for statistically singular models such as deep neural networks (DNNs) is difficult due to their complex nature. This study focuses on Takeuchi's information criterion (TIC) to investigate the conditions under which this classical measure can effectively explain the generalization gaps of DNNs. Importantly, the developed theory indicates the applicability of TIC near the neural tangent kernel (NTK) regime. In a series of experiments, we trained more than 5,000 DNN models with 12 architectures, including large models (e.g., VGG-16), on four datasets, and estimated the corresponding TIC values to examine the relationship between the generalization gap and the TIC estimates. We applied several TIC approximation methods with feasible computational costs and assessed the accuracy trade-off. Our experimental results indicate that the estimated TIC values correlate well with the generalization gap under conditions close to the NTK regime. However, we show both theoretically and empirically that outside the NTK regime such correlation disappears. Finally, we demonstrate that TIC provides better trial pruning ability than existing methods for hyperparameter optimization.