Cross-regularization: Adaptive Model Complexity through Validation Gradients

TL;DR

Cross-regularization introduces a gradient-based method to adapt regularization parameters during training, balancing model complexity and overfitting by leveraging validation data, with applications in neural networks and data augmentation.

Contribution

It presents a novel adaptive regularization technique that uses validation gradients to automatically tune regularization parameters during training.

Findings

High noise tolerance observed in neural network regularization.

Emergent architecture-specific regularization patterns.

Seamless integration with data augmentation and uncertainty calibration.

Abstract

Model regularization requires extensive manual tuning to balance complexity against overfitting. Cross-regularization resolves this tradeoff by directly adapting regularization parameters through validation gradients during training. The method splits parameter optimization - training data guides feature learning while validation data shapes complexity controls - converging provably to cross-validation optima. When implemented through noise injection in neural networks, this approach reveals striking patterns: unexpectedly high noise tolerance and architecture-specific regularization that emerges organically during training. Beyond complexity control, the framework integrates seamlessly with data augmentation, uncertainty calibration and growing datasets while maintaining single-run efficiency through a simple gradient-based approach.