Implementing Adaptations for Vision AutoRegressive Model

TL;DR

This paper explores how to adapt Vision AutoRegressive models for specific tasks, compares them with diffusion models, and highlights the need for better private adaptation techniques for VAR.

Contribution

It implements and benchmarks various adaptation strategies for VAR, comparing them to diffusion models, and identifies challenges in private adaptations.

Findings

VAR outperforms DMs in non-private settings

DP adaptations for VAR underperform, indicating need for further research

Benchmarking provides insights into adaptation strategies for VAR

Abstract

Vision AutoRegressive model (VAR) was recently introduced as an alternative to Diffusion Models (DMs) in image generation domain. In this work we focus on its adaptations, which aim to fine-tune pre-trained models to perform specific downstream tasks, like medical data generation. While for DMs there exist many techniques, adaptations for VAR remain underexplored. Similarly, differentially private (DP) adaptations-ones that aim to preserve privacy of the adaptation data-have been extensively studied for DMs, while VAR lacks such solutions. In our work, we implement and benchmark many strategies for VAR, and compare them to state-of-the-art DM adaptation strategies. We observe that VAR outperforms DMs for non-DP adaptations, however, the performance of DP suffers, which necessitates further research in private adaptations for VAR. Code is available at https://github.com/sprintml/finetuning_var_dp.