Representation Learning with Adaptive Superpixel Coding

TL;DR

This paper introduces Adaptive Superpixel Coding (ASC), a self-supervised Transformer-based model that dynamically adjusts to image content, overcoming fixed patch limitations and outperforming traditional methods on standard benchmarks.

Contribution

The paper presents a novel adaptive superpixel layer for Transformers, enabling content-aware image partitioning in self-supervised learning.

Findings

ASC outperforms traditional fixed-patch models on image benchmarks.

Adaptive superpixel layers improve model flexibility and effectiveness.

The approach overcomes limitations of fixed-size patch partitioning.

Abstract

Deep learning vision models are typically tailored for specific modalities and often rely on domain-specific assumptions, such as the grid structures used by nearly all existing vision models. In this work, we propose a self-supervised model based on Transformers, which we call Adaptive Superpixel Coding (ASC). The key insight of our model is to overcome the limitations of traditional Vision Transformers, which depend on fixed-size and non-adaptive patch partitioning. Instead, ASC employs adaptive superpixel layers that dynamically adjust to the underlying image content. We analyze key properties of the approach that make it effective, and find that our method outperforms widely-used alternatives on standard image downstream task benchmarks.