The Influences of Color and Shape Features in Visual Contrastive Learning

TL;DR

This paper investigates how color and shape features influence visual contrastive learning, revealing that contrastive representations favor color clustering over shape, with data augmentation playing a significant role in these effects.

Contribution

It introduces a quantitative evaluation of color and shape contributions in contrastive learning using novel metrics and ablation experiments.

Findings

Contrastive representations cluster more by color than shape.

Contrastive models contain less shape information than supervised models.

Data augmentation significantly affects feature representation in contrastive learning.

Abstract

In the field of visual representation learning, performance of contrastive learning has been catching up with the supervised method which is commonly a classification convolutional neural network. However, most of the research work focuses on improving the accuracy of downstream tasks such as image classification and object detection. For visual contrastive learning, the influences of individual image features (e.g., color and shape) to model performance remain ambiguous. This paper investigates such influences by designing various ablation experiments, the results of which are evaluated by specifically designed metrics. While these metrics are not invented by us, we first use them in the field of representation evaluation. Specifically, we assess the contribution of two primary image features (i.e., color and shape) in a quantitative way. Experimental results show that compared with supervised representations, contrastive representations tend to cluster with objects of similar color in the representation space, and contain less shape information than supervised representations. Finally, we discuss that the current data augmentation is responsible for these results. We believe that exploring an unsupervised augmentation method that