Data-driven Regularization via Racecar Training for Generalizing Neural Networks

TL;DR

This paper introduces Racecar training, a data-dependent orthogonality regularization method that enhances neural network generalization by promoting more balanced and explainable internal representations, leading to improved task performance.

Contribution

It presents a novel, practical training method that enforces data-dependent orthogonality constraints via a reverse pass, improving neural network generalization and interpretability.

Findings

Networks trained with Racecar show more balanced mutual information across layers.

Racecar training leads to improved performance on various tasks.

The method enhances explainability and feature generality in neural networks.

Abstract

We propose a novel training approach for improving the generalization in neural networks. We show that in contrast to regular constraints for orthogonality, our approach represents a {\em data-dependent} orthogonality constraint, and is closely related to singular value decompositions of the weight matrices. We also show how our formulation is easy to realize in practical network architectures via a reverse pass, which aims for reconstructing the full sequence of internal states of the network. Despite being a surprisingly simple change, we demonstrate that this forward-backward training approach, which we refer to as {\em racecar} training, leads to significantly more generic features being extracted from a given data set. Networks trained with our approach show more balanced mutual information between input and output throughout all layers, yield improved explainability and, exhibit improved performance for a variety of tasks and task transfers.