Revisiting Autoregressive Models for Generative Image Classification

TL;DR

This paper improves autoregressive generative image classifiers by using order-marginalized predictions, leading to better accuracy and efficiency compared to diffusion models and competitive performance with discriminative models.

Contribution

It introduces an order-marginalization technique for AR models, overcoming fixed token order limitations and enhancing classification performance and efficiency.

Findings

Outperforms diffusion-based classifiers on multiple benchmarks.

Achieves up to 25x higher efficiency.

Delivers competitive results with self-supervised discriminative models.

Abstract

Class-conditional generative models have emerged as accurate and robust classifiers, with diffusion models demonstrating clear advantages over other visual generative paradigms, including autoregressive (AR) models. In this work, we revisit visual AR-based generative classifiers and identify an important limitation of prior approaches: their reliance on a fixed token order, which imposes a restrictive inductive bias for image understanding. We observe that single-order predictions rely more on partial discriminative cues, while averaging over multiple token orders provides a more comprehensive signal. Based on this insight, we leverage recent any-order AR models to estimate order-marginalized predictions, unlocking the high classification potential of AR models. Our approach consistently outperforms diffusion-based classifiers across diverse image classification benchmarks, while being up to 25x more efficient. Compared to state-of-the-art self-supervised discriminative models, our method delivers competitive classification performance - a notable achievement for generative classifiers.