Contrastive Learning for Lifted Networks

TL;DR

This paper introduces a contrastive learning approach for lifted neural networks, overcoming limitations of previous methods, and demonstrates it approximates back-propagation effectively, enabling efficient training on parallel hardware.

Contribution

The paper proposes a novel contrastive loss for lifted networks, improving training effectiveness and theoretical approximation to back-propagation.

Findings

Contrastive training outperforms traditional lifted network training methods.

The proposed approach approximates back-propagation both theoretically and practically.

Enhanced training efficiency on parallel hardware.

Abstract

In this work we address supervised learning of neural networks via lifted network formulations. Lifted networks are interesting because they allow training on massively parallel hardware and assign energy models to discriminatively trained neural networks. We demonstrate that the training methods for lifted networks proposed in the literature have significant limitations and show how to use a contrastive loss to address those limitations. We demonstrate that this contrastive training approximates back-propagation in theory and in practice and that it is superior to the training objective regularly used for lifted networks.