ADDP: Learning General Representations for Image Recognition and Generation with Alternating Denoising Diffusion Process

TL;DR

ADDP introduces a unified framework that learns general representations for both image recognition and generation by alternating between pixel decoding and VQ token generation, achieving state-of-the-art results in multiple tasks.

Contribution

The paper proposes the first unified approach integrating pixel-based recognition and VQ token-based generation within a single diffusion process.

Findings

Achieves competitive results on ImageNet classification.

Demonstrates strong performance on COCO detection.

Excels in ADE20k segmentation tasks.

Abstract

Image recognition and generation have long been developed independently of each other. With the recent trend towards general-purpose representation learning, the development of general representations for both recognition and generation tasks is also promoted. However, preliminary attempts mainly focus on generation performance, but are still inferior on recognition tasks. These methods are modeled in the vector-quantized (VQ) space, whereas leading recognition methods use pixels as inputs. Our key insights are twofold: (1) pixels as inputs are crucial for recognition tasks; (2) VQ tokens as reconstruction targets are beneficial for generation tasks. These observations motivate us to propose an Alternating Denoising Diffusion Process (ADDP) that integrates these two spaces within a single representation learning framework. In each denoising step, our method first decodes pixels from previous VQ tokens, then generates new VQ tokens from the decoded pixels. The diffusion process gradually masks out a portion of VQ tokens to construct the training samples. The learned representations can be used to generate diverse high-fidelity images and also demonstrate excellent transfer performance on recognition tasks. Extensive experiments show that our method achieves competitive performance on unconditional generation, ImageNet classification, COCO detection, and ADE20k segmentation. Importantly, our method represents the first successful development of general representations applicable to both generation and dense recognition tasks. Code is released at \url{https://github.com/ChangyaoTian/ADDP}.