Grid Partitioned Attention: Efficient TransformerApproximation with Inductive Bias for High Resolution Detail Generation

TL;DR

This paper introduces Grid Partitioned Attention (GPA), an efficient approximate attention mechanism for high-resolution image generation that reduces memory usage by focusing on local spatial correlations, enabling new architectures and state-of-the-art results.

Contribution

The paper proposes GPA, a novel attention layer with a sparse inductive bias, and demonstrates its effectiveness in high-resolution image generation and pose morphing tasks.

Findings

Achieves state-of-the-art results in human pose morphing benchmarks.

Reduces memory requirements for high-resolution image generation.

Enables new deep learning architectures with copying modules.

Abstract

Attention is a general reasoning mechanism than can flexibly deal with image information, but its memory requirements had made it so far impractical for high resolution image generation. We present Grid Partitioned Attention (GPA), a new approximate attention algorithm that leverages a sparse inductive bias for higher computational and memory efficiency in image domains: queries attend only to few keys, spatially close queries attend to close keys due to correlations. Our paper introduces the new attention layer, analyzes its complexity and how the trade-off between memory usage and model power can be tuned by the hyper-parameters.We will show how such attention enables novel deep learning architectures with copying modules that are especially useful for conditional image generation tasks like pose morphing. Our contributions are (i) algorithm and code1of the novel GPA layer, (ii) a novel deep attention-copying architecture, and (iii) new state-of-the art experimental results in human pose morphing generation benchmarks.