Contrasting quadratic assignments for set-based representation learning

TL;DR

This paper introduces a set-based contrastive learning approach using quadratic assignment theory to better capture intra- and inter-set similarities, enhancing representation quality.

Contribution

It proposes a novel set-contrastive objective that extends traditional pairwise contrastive learning by considering sets of views, improving learned representations.

Findings

Improved performance in metric learning tasks

Enhanced self-supervised classification accuracy

Set-based contrastive method outperforms pairwise approaches

Abstract

The standard approach to contrastive learning is to maximize the agreement between different views of the data. The views are ordered in pairs, such that they are either positive, encoding different views of the same object, or negative, corresponding to views of different objects. The supervisory signal comes from maximizing the total similarity over positive pairs, while the negative pairs are needed to avoid collapse. In this work, we note that the approach of considering individual pairs cannot account for both intra-set and inter-set similarities when the sets are formed from the views of the data. It thus limits the information content of the supervisory signal available to train representations. We propose to go beyond contrasting individual pairs of objects by focusing on contrasting objects as sets. For this, we use combinatorial quadratic assignment theory designed to evaluate set and graph similarities and derive set-contrastive objective as a regularizer for contrastive learning methods. We conduct experiments and demonstrate that our method improves learned representations for the tasks of metric learning and self-supervised classification.