Investigating Permutation-Invariant Discrete Representation Learning for Spatially Aligned Images

TL;DR

This paper introduces a permutation-invariant vector-quantized autoencoder that captures global semantic features of images without positional information, enabling efficient image interpolation and synthesis.

Contribution

It proposes the PI-VQ model with a novel matching quantization algorithm, improving capacity and enabling position-free, interpretable image representations.

Findings

PI-VQ captures global semantic features without positional info.

Matching quantization increases effective capacity by 3.5×.

The approach achieves competitive metrics on CelebA datasets.

Abstract

Vector quantization approaches (VQ-VAE, VQ-GAN) learn discrete neural representations of images, but these representations are inherently position-dependent: codes are spatially arranged and contextually entangled, requiring autoregressive or diffusion-based priors to model their dependencies at sample time. In this work, we ask whether positional information is necessary for discrete representations of spatially aligned data. We propose the permutation-invariant vector-quantized autoencoder (PI-VQ), in which latent codes are constrained to carry no positional information. We find that this constraint encourages codes to capture global, semantic features, and enables direct interpolation between images without a learned prior. To address the reduced information capacity of permutation-invariant representations, we introduce matching quantization, a vector quantization algorithm based on optimal bipartite matching that increases effective bottleneck capacity by $3.5\times$ relative to naive nearest-neighbour quantization. The compositional structure of the learned codes further enables interpolation-based sampling, allowing synthesis of novel images in a single forward pass. We evaluate PI-VQ on CelebA, CelebA-HQ and FFHQ, obtaining competitive precision, density and coverage metrics for images synthesised with our approach. We discuss the trade-offs inherent to position-free representations, including separability and interpretability of the latent codes, pointing to numerous directions for future work.