Mapping Networks

TL;DR

Mapping Networks propose a low-dimensional latent space approach to replace high-dimensional weights in deep learning, reducing overfitting and maintaining or improving performance across various tasks with significantly fewer parameters.

Contribution

The paper introduces Mapping Networks, a novel method that maps a low-dimensional latent space to the weight space, supported by a theoretical Mapping Theorem, to improve efficiency and reduce overfitting.

Findings

Achieves around 500x reduction in trainable parameters.

Maintains or surpasses target network performance on vision and sequence tasks.

Effectively reduces overfitting in large-scale models.

Abstract

The escalating parameter counts in modern deep learning models pose a fundamental challenge to efficient training and resolution of overfitting. We address this by introducing the \emph{Mapping Networks} which replace the high dimensional weight space by a compact, trainable latent vector based on the hypothesis that the trained parameters of large networks reside on smooth, low-dimensional manifolds. Henceforth, the Mapping Theorem enforced by a dedicated Mapping Loss, shows the existence of a mapping from this latent space to the target weight space both theoretically and in practice. Mapping Networks significantly reduce overfitting and achieve comparable to better performance than target network across complex vision and sequence tasks, including Image Classification, Deepfake Detection etc, with $\mathbf{99.5\%}$, i.e., around $500\times$ reduction in trainable parameters.