Vector Contrastive Learning For Pixel-Wise Pretraining In Medical Vision

TL;DR

This paper introduces a novel vector contrastive learning approach called COVER for pixel-wise pretraining in medical vision, effectively preserving feature correlations and improving model performance across multiple tasks and modalities.

Contribution

It reformulates contrastive learning as a vector regression problem and proposes the COVER framework to enhance pixel-wise self-supervised pretraining in medical imaging.

Findings

COVER significantly improves pixel-wise SSP performance.

It demonstrates superior results across 8 diverse medical imaging tasks.

The approach enhances generalizability of medical visual foundation models.

Abstract

Contrastive learning (CL) has become a cornerstone of self-supervised pretraining (SSP) in foundation models, however, extending CL to pixel-wise representation, crucial for medical vision, remains an open problem. Standard CL formulates SSP as a binary optimization problem (binary CL) where the excessive pursuit of feature dispersion leads to an over-dispersion problem, breaking pixel-wise feature correlation thus disrupting the intra-class distribution. Our vector CL reformulates CL as a vector regression problem, enabling dispersion quantification in pixel-wise pretraining via modeling feature distances in regressing displacement vectors. To implement this novel paradigm, we propose the COntrast in VEctor Regression (COVER) framework. COVER establishes an extendable vector-based self-learning, enforces a consistent optimization flow from vector regression to distance modeling, and leverages a vector pyramid architecture for granularity adaptation, thus preserving pixel-wise feature correlations in SSP. Extensive experiments across 8 tasks, spanning 2 dimensions and 4 modalities, show that COVER significantly improves pixel-wise SSP, advancing generalizable medical visual foundation models.