Natural Image Manipulation for Autoregressive Models Using Fisher Scores

TL;DR

This paper introduces a method using Fisher scores to enable meaningful image manipulations in autoregressive models, bridging the gap with latent variable models for controlled sample generation.

Contribution

It proposes a novel approach to extract embeddings from autoregressive models via Fisher scores, facilitating semantic manipulations similar to latent variable models.

Findings

Fisher scores provide more meaningful embeddings for image manipulation.

The method improves controllability of sample generation in autoregressive models.

Compared to network activations, Fisher scores yield better semantic interpolation.

Abstract

Deep autoregressive models are one of the most powerful models that exist today which achieve state-of-the-art bits per dim. However, they lie at a strict disadvantage when it comes to controlled sample generation compared to latent variable models. Latent variable models such as VAEs and normalizing flows allow meaningful semantic manipulations in latent space, which autoregressive models do not have. In this paper, we propose using Fisher scores as a method to extract embeddings from an autoregressive model to use for interpolation and show that our method provides more meaningful sample manipulation compared to alternate embeddings such as network activations.