Semantic-Aware Autoregressive Image Modeling for Visual Representation Learning

TL;DR

This paper introduces SemAIM, a semantic-aware autoregressive image modeling approach that improves self-supervised visual representation learning by modeling images from semantic to less semantic patches, achieving state-of-the-art results.

Contribution

SemAIM proposes a novel permutation-based autoregressive modeling method guided by semantic similarity, enhancing high-level feature learning in images.

Findings

Achieves 84.1% top-1 accuracy on ImageNet with ViT-B.

Outperforms vanilla MAE in object detection and segmentation tasks.

Demonstrates state-of-the-art performance across multiple downstream tasks.

Abstract

The development of autoregressive modeling (AM) in computer vision lags behind natural language processing (NLP) in self-supervised pre-training. This is mainly caused by the challenge that images are not sequential signals and lack a natural order when applying autoregressive modeling. In this study, inspired by human beings' way of grasping an image, i.e., focusing on the main object first, we present a semantic-aware autoregressive image modeling (SemAIM) method to tackle this challenge. The key insight of SemAIM is to autoregressive model images from the semantic patches to the less semantic patches. To this end, we first calculate a semantic-aware permutation of patches according to their feature similarities and then perform the autoregression procedure based on the permutation. In addition, considering that the raw pixels of patches are low-level signals and are not ideal prediction targets for learning high-level semantic representation, we also explore utilizing the patch features as the prediction targets. Extensive experiments are conducted on a broad range of downstream tasks, including image classification, object detection, and instance/semantic segmentation, to evaluate the performance of SemAIM. The results demonstrate SemAIM achieves state-of-the-art performance compared with other self-supervised methods. Specifically, with ViT-B, SemAIM achieves 84.1% top-1 accuracy for fine-tuning on ImageNet, 51.3% AP and 45.4% AP for object detection and instance segmentation on COCO, which outperforms the vanilla MAE by 0.5%, 1.0%, and 0.5%, respectively.