SVMax: A Feature Embedding Regularizer

TL;DR

SVMax is a novel regularizer that enhances neural network performance by promoting uniform feature embeddings, applicable in both supervised and unsupervised learning, and improves stability and results in retrieval and generative tasks.

Contribution

We introduce SVMax, a new regularizer that maximizes singular values of feature embeddings to improve network regularization and performance.

Findings

SVMax improves retrieval accuracy across multiple ranking losses.

SVMax supports larger learning rates and mitigates model collapse.

SVMax enhances unsupervised embedding quality on Gaussian mixtures.

Abstract

A neural network regularizer (e.g., weight decay) boosts performance by explicitly penalizing the complexity of a network. In this paper, we penalize inferior network activations -- feature embeddings -- which in turn regularize the network's weights implicitly. We propose singular value maximization (SVMax) to learn a more uniform feature embedding. The SVMax regularizer supports both supervised and unsupervised learning. Our formulation mitigates model collapse and enables larger learning rates. We evaluate the SVMax regularizer using both retrieval and generative adversarial networks. We leverage a synthetic mixture of Gaussians dataset to evaluate SVMax in an unsupervised setting. For retrieval networks, SVMax achieves significant improvement margins across various ranking losses. Code available at https://bit.ly/3jNkgDt