Proximal Mapping for Deep Regularization

TL;DR

This paper introduces a novel deep learning regularization technique by inserting proximal mapping layers that directly regularize hidden layer outputs, leading to improved robustness and performance in temporal learning and multiview modeling.

Contribution

It proposes a new method of regularization in deep networks using proximal mapping layers, directly targeting hidden layer outputs, which is a departure from traditional weight-based regularization.

Findings

Outperforms state-of-the-art methods in robust temporal learning.

Enhances multiview modeling accuracy.

Connects to kernel warping and dropout techniques.

Abstract

Underpinning the success of deep learning is effective regularizations that allow a variety of priors in data to be modeled. For example, robustness to adversarial perturbations, and correlations between multiple modalities. However, most regularizers are specified in terms of hidden layer outputs, which are not themselves optimization variables. In contrast to prevalent methods that optimize them indirectly through model weights, we propose inserting proximal mapping as a new layer to the deep network, which directly and explicitly produces well regularized hidden layer outputs. The resulting technique is shown well connected to kernel warping and dropout, and novel algorithms were developed for robust temporal learning and multiview modeling, both outperforming state-of-the-art methods.