Towards noise contrastive estimation with soft targets for conditional models

TL;DR

This paper introduces a new loss function called soft target InfoNCE, compatible with probabilistic targets, that improves training of deep neural networks by combining the benefits of noise contrastive estimation and soft targets.

Contribution

The authors propose a novel loss function that integrates soft targets with InfoNCE, addressing limitations of standard contrastive methods and broadening their applicability in supervised learning.

Findings

Soft target InfoNCE matches strong soft target cross-entropy baselines.

It outperforms hard target NLL and InfoNCE on benchmarks like ImageNet.

The method is simple, efficient, and compatible with existing deep classification models.

Abstract

Soft targets combined with the cross-entropy loss have shown to improve generalization performance of deep neural networks on supervised classification tasks. The standard cross-entropy loss however assumes data to be categorically distributed, which may often not be the case in practice. In contrast, InfoNCE does not rely on such an explicit assumption but instead implicitly estimates the true conditional through negative sampling. Unfortunately, it cannot be combined with soft targets in its standard formulation, hindering its use in combination with sophisticated training strategies. In this paper, we address this limitation by proposing a loss function that is compatible with probabilistic targets. Our new soft target InfoNCE loss is conceptually simple, efficient to compute, and can be motivated through the framework of noise contrastive estimation. Using a toy example, we demonstrate shortcomings of the categorical distribution assumption of cross-entropy, and discuss implications of sampling from soft distributions. We observe that soft target InfoNCE performs on par with strong soft target cross-entropy baselines and outperforms hard target NLL and InfoNCE losses on popular benchmarks, including ImageNet. Finally, we provide a simple implementation of our loss, geared towards supervised classification and fully compatible with deep classification models trained with cross-entropy.