DIP: Unsupervised Dense In-Context Post-training of Visual Representations

TL;DR

DIP is an unsupervised post-training method that enhances dense visual representations by simulating in-context tasks, significantly improving downstream scene understanding performance with minimal computational resources.

Contribution

It introduces a simple, unsupervised post-training approach using pseudo in-context tasks generated by a diffusion model, avoiding complex architectures.

Findings

Outperforms initial vision encoders on downstream tasks

Requires less than 9 hours of training on a single GPU

Effective for various real-world scene understanding tasks

Abstract

We introduce DIP, a novel unsupervised post-training method designed to enhance dense image representations in large-scale pretrained vision encoders for in-context scene understanding. Unlike prior approaches that rely on complex self-distillation architectures, our method trains the vision encoder using pseudo-tasks that explicitly simulate downstream in-context scenarios, inspired by meta-learning principles. To enable post-training on unlabeled data, we propose an automatic mechanism for generating in-context tasks that combines a pretrained diffusion model and the vision encoder itself. DIP is simple, unsupervised, and computationally efficient, requiring less than 9 hours on a single A100 GPU. By learning dense representations through pseudo in-context tasks, it achieves strong performance across a wide variety of downstream real-world in-context scene understanding tasks. It outperforms both the initial vision encoder and prior methods, offering a practical and effective solution for improving dense representations. Code available here: https://github.com/sirkosophia/DIP