Self-Supervised Representation Learning with Cross-Context Learning between Global and Hypercolumn Features

TL;DR

This paper introduces a novel self-supervised learning framework called CGH that enhances representation learning by enforcing consistency between global and hypercolumn features, improving performance on classification tasks.

Contribution

The paper proposes a new cross-context learning method that aligns global and intermediate features, extending contrastive learning with inter-layer relation guidance.

Findings

Outperforms state-of-the-art methods on classification benchmarks

Improves the capture of inter-instance relationships

Enhances feature representations with cross-context consistency

Abstract

Whilst contrastive learning yields powerful representations by matching different augmented views of the same instance, it lacks the ability to capture the similarities between different instances. One popular way to address this limitation is by learning global features (after the global pooling) to capture inter-instance relationships based on knowledge distillation, where the global features of the teacher are used to guide the learning of the global features of the student. Inspired by cross-modality learning, we extend this existing framework that only learns from global features by encouraging the global features and intermediate layer features to learn from each other. This leads to our novel self-supervised framework: cross-context learning between global and hypercolumn features (CGH), that enforces the consistency of instance relations between low- and high-level semantics. Specifically, we stack the intermediate feature maps to construct a hypercolumn representation so that we can measure instance relations using two contexts (hypercolumn and global feature) separately, and then use the relations of one context to guide the learning of the other. This cross-context learning allows the model to learn from the differences between the two contexts. The experimental results on linear classification and downstream tasks show that our method outperforms the state-of-the-art methods.