Modulated Contrast for Versatile Image Synthesis

TL;DR

MoNCE introduces an adaptive contrastive metric with optimal transport to improve image similarity measurement, reducing artifacts and enhancing performance across diverse image synthesis tasks.

Contribution

The paper proposes MoNCE, a novel contrastive metric that adaptively weights negative samples and uses optimal transport for better image similarity assessment.

Findings

MoNCE outperforms existing metrics in multiple image translation tasks.

Adaptive negative sampling improves contrastive learning effectiveness.

Optimal transport enhances collaboration across multiple contrastive objectives.

Abstract

Perceiving the similarity between images has been a long-standing and fundamental problem underlying various visual generation tasks. Predominant approaches measure the inter-image distance by computing pointwise absolute deviations, which tends to estimate the median of instance distributions and leads to blurs and artifacts in the generated images. This paper presents MoNCE, a versatile metric that introduces image contrast to learn a calibrated metric for the perception of multifaceted inter-image distances. Unlike vanilla contrast which indiscriminately pushes negative samples from the anchor regardless of their similarity, we propose to re-weight the pushing force of negative samples adaptively according to their similarity to the anchor, which facilitates the contrastive learning from informative negative samples. Since multiple patch-level contrastive objectives are involved in image distance measurement, we introduce optimal transport in MoNCE to modulate the pushing force of negative samples collaboratively across multiple contrastive objectives. Extensive experiments over multiple image translation tasks show that the proposed MoNCE outperforms various prevailing metrics substantially. The code is available at https://github.com/fnzhan/MoNCE.